0 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



© Publication number: 



0 626 635 A2 



EUROPEAN PATENT APPLICATION 



@ Application number: 94302626.0 
© Date of filing: 13.04.94 



© Int. CI. 5 : G06F 3/00 



® 


Priority: 24.05.93 US 67574 


Inventor: Palrang, Joseph M. 






701 San Conrado Terrace, Nr. 2 


© 


Date of publication of application: 


Sunnyvale, California 94086 (US) 




ou.i i .y4 ounetin y4/4o 


Inventor: Gosling, James A. 






P.O.Box 620509 


© 


Designated Contracting States: 


Woodside,CA.94062 (US) 




DE FR GB NL 


Inventor: Frank, Edward H. 






127 Brookside Road 


© 


Applicant: FIRSTPERSON, INC. 


Portola Valley, California 94028 (US) 




100 Hamilton Avenue 


Inventor: LaVallee, David A. 




Palo Alto, California 94301 (US) 


651 Seale Avenue 






Palo Alto, California 94301 (US) 


© 


Inventor: Naughton, Patrick J. 


Inventor: Sheridan, R. Michael 




652 Loma Verde Avenue 


10651 Oakton Ridge Ct. 




Palo Alto, California 94036 (US) 


Oakton, Virginia 22124 (US) 




Inventor: Warth, Chris 






1617 Dolores Street 






San Francisco, California 94110 (US) 


© Representative: Wombwell, Francis 




Inventor: Clanton, Charles H., Ill 


Potts, Kerr & Co. 




220 Downey Street 


15, Hamilton Square 




San Francisco, California 94117 (US) 


Birkenhead Merseyside L41 6BR (GB) 



© Improved graphical user interface with method and apparatus for interfacing to remote devices. 



Csl 
< 

in 

CO 
CO 

<D 
<M 
CO 



Q. 

UJ 



© An intuitive graphical user interface is based 
upon a geographic map structure, and includes a 
system for controlling remote external electronic de- 
vices. In the defined graphical user interface, each 
Space of the geographic map structure is rendered 
on a touch screen display as a graphic image of a 
geographic space. Within each Space are colored 
cartoon-like icons called "objects" which can be 
Selected and manipulated by the user. Certain ob- 
jects, referred to as portals, transport the user from 
one Space to another Space when Selected. Other 
objects, referred to as buttons, perform associated 
actions or functions when Selected. The graphical 
user interface is displayed on a hand-held display 
device used to control remote devices. Each remote 
electronic device transmits a user interface program 
object that defines a graphical user interface to the 
display device. The hand-held display devices dis- 
plays the graphical user interface associated with the 
remote device when a user selects an icon asso- 
ciated with the remote device on the display de- 



vice's touch screen display. 




Figure la 



Rank Xerox (UK) Business Services 

(3. 10/3.09/3.3.4) 



EP 0 626 635 A2 



BACKGROUND OF THE INVENTION 

1. Field of the Invention: 

The present invention relates to the field of 
graphical user interfaces for computer systems 
and, more particularly, to a method and apparatus 
for providing an intuitive methodology for an oper- 
ator to control remote devices with a computer 
controlled object-oriented user interface utilizing 
animated graphic images. 

2. Art Background: 

Most popular graphical user interfaces that are 
available today, such as the Xerox Star®, Apple 
Macintosh®, and Microsoft Windows graphical user 
interfaces, are based upon the "desktop meta- 
phor." In a desktop metaphor system, the display 
screen is treated as a virtual desktop. Graphical 
symbols placed on the virtual desktop (i.e. the 
display screen) are used to represent common 
objects found in an office environment such as 
files, file folders, and printers. These desktop meta- 
phor systems have several limitations which ham- 
per the effectiveness of the man-machine interface 
between the human operator and the computer 
system. 

One problem with the existing desktop meta- 
phor graphical user interfaces is the use of a hier- 
archical file system. Most desktop metaphor 
graphical user interfaces are based upon a com- 
puter having a file system in which the computer 
files are organized in a hierarchical structure. The 
hierarchical file system structure has a root direc- 
tory and several sub-directories within the root di- 
rectory. Each sub-directory may have sub-sub-di- 
rectories, and so on. The hierarchical file system 
structure is graphically illustrated on the screen as 
some type of root file cabinet with file folders 
inside. The Microsoft Windows and Apple Macin- 
tosh® operating systems both use hierarchical file 
systems illustrated as file folders. 

The hierarchical structure of file folders is often 
very confusing to new users. For example, many 
users get lost while navigating through hierarchical 
file systems because the system of keeping files in 
file folders, which are in other file folders nested to 
an arbitrary depth is inherently confusing. Users 
become lost because each file folder looks like 
every other file folder, and the user is provided with 
little feedback as to his location within the file 
system hierarchy. Also, each file folder provides no 
information as to what the file folder contains other 
than a user-defined label. For example, except for 
a short text label, which may have been defined by 
another user, the user generally is not able to 
determine whether an open folder is an accounting 



file folder containing spreadsheet data, or a data 
base file folder containing a customer data base. 
Therefore, navigating through the complex file sys- 
tem hierarchy is difficult since the current desktop 
5 metaphor graphical user interface offers little feed- 
back as to a user's location within the file hierar- 
chy. 

In current graphical user interface systems 
based upon the desktop metaphor, certain applica- 

w tions often do not fit well into a desktop metaphor 
environment conceptually. Specifically, non-office 
related applications may be inappropriate for a 
system based upon the desktop metaphor. For 
example, in a desktop metaphor, an application that 

15 controls a television would have to be placed into a 
"paper" file folder on the desktop. 

Another common problem with desktop meta- 
phor user interfaces is that displaying a large num- 
ber of applications on the desktop results in a 

20 cluttered desktop. Many users of desktop metaphor 
systems keep large numbers of applications and 
files, usually in icon form, on the display screen. 
Some attempt is made to use a graphical repre- 
sentation that identifies the type of application or 

25 file. However, when a user wishes to open a par- 
ticular application or file, the user must search the 
display screen to find the desired icon just as if the 
user were trying to find a particular paper file on a 
messy desktop. Icons placed the desktop have no 

30 apparent hierarchy, and are all equal. This equality 
may confuse a user, and may create problems 
when the user wants to switch from one task to 
another. 

Several systems have evolved to improve the 

35 organization of the desktop metaphor graphical 
user interface. One system is the concept of 
"Rooms". See, Patrick Chan, "Learning Consid- 
erations in User Interface Design: The Room 
Model", Report CS-84-16, University of Waterloo 

40 Computer Science Department, Ontario, Canada, 
July, 1984; Brenda Laurel, "Computers as The- 
atre", Addison-Wesley Publishing Co., 1991; Bren- 
da Laurel, "The Art of Human-Computer Interface 
Design", Addison-Wesley Publishing Co., 1990; 

45 U.S. Patent 5,072, 412, Henderson, Jr. et al. f "User 
Interface with Multiple Workspaces for Sharing 
Display System Objects", and the references 
cited therein. 

In systems based upon the Rooms concept, 

50 the user creates different rooms where the user 
performs a single task or a group of related tasks. 
Each room represents an individual virtual desktop 
with a group of associated tasks. Each room is 
given a label that identifies the room with the 

55 room's functions. For example, a user may create a 
mail room, a typing room with a word processor, 
and an accounting room with a spreadsheet. Each 
room has only those icons and windows that are 
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associated with the particular task of the room, as 
well as any windows shared by all rooms, such as 
a clock. 

To switch between tasks in a system that uses 
the Rooms concept, a user moves between the 
different rooms that are associated with the dif- 
ferent tasks. For example, to go from a spread- 
sheet to retrieve email, a user moves from the 
accounting room to the mail room. To move be- 
tween the two rooms, a user selects an icon that 
represents a door connecting the current room to 
the other room. Users can create doors between 
any of the available rooms. 

Although the Rooms concept helps solve the 
problem of the "cluttered desktop," the Rooms 
concept still relies upon the common desktop met- 
aphor. For example, the Rooms interface concept 
does not help simplify the confusing hierarchical 
file structure used to store information. The Rooms 
concept also does little to solve the form and 
function inconsistencies that are apparent when 
applications which do not fit into the desktop meta- 
phor are added to the system. 

In prior art graphical user interfaces, each time 
a new piece of hardware is introduced to the sys- 
tem a new piece of control software must also be 
added by the user. For example, if a user of an 
Apple® Macintosh® or a user of Microsoft® Win- 
dows desires to add a new printer to the system, 
the user must also install a new printer driver to 
control the added printer. Therefore, each time a 
user desires to add a new device to the system, 
the user must also follow a complicated procedure 
to add the appropriate software. 

As will be described, the present invention 
provides an improved graphical user interface that 
overcomes several of the problems associated with 
desktop metaphor systems. A method and appara- 
tus for obtaining the user interface of a remote 
device without requiring the user to load the related 
software is also disclosed. The remote device can 
be controlled using the remote device's user inter- 
face as displayed on the present invention's dis- 
play screen. Although the user interface is de- 
scribed with reference to the control of remote 
devices, it will be appreciated from the discussion 
that follows that the user interface may be applied 
to a variety of applications in computer worksta- 
tions, portable computers, and hand held control 
systems. 

SUMMARY OF THE INVENTION 

The present invention provides an intuitive 
graphical user interface and a method and appara- 
tus for controlling remote devices. The graphical 
user interface of the present invention is displayed 
on a hand-held display device. The hand-held dis- 



play device comprises at least one central process- 
ing unit (CPU) coupled to a graphic display sys- 
tem. In the present embodiment, the graphic dis- 
play system is touch sensitive to permit the user to 

5 input information using a fingertip without the need 
for additional input devices such as a keyboard. A 
communications circuit is also coupled to the CPU 
transmitting information over some communication 
medium. The communications circuit is used to 

w control and to communicate with remote devices. 

The graphical user interface of the present 
invention creates a world of Spaces organized into 
a geographic map structure that is simple to navi- 
gate through since each Space provides familiar 

/5 geographic surroundings. Each Space is represent- 
ed by a background image which provides naviga- 
tional landmarks that identify where the Space is 
located within the geographic map structure. For 
example, the background image of a living room 

20 Space contains items typically found in a living 
room such as a lamp, a chair, and a table. There- 
fore, navigation through this "world" is made easier 
because the user can orient himself by looking 
around and examining the surroundings just as a 

25 person would do in the real world. 

Within each Space, the user interface provides 
graphic objects that can be selected and manipu- 
lated by the user. The graphic objects are rendered 
as animated cartoon-like graphic objects that are 

30 easily identifiable as real world objects such as 
televisions and VCRs. There are at least three 
types of objects that exist in the Spaces of the user 
interface: data objects, buttons, and portals. Data 
objects are objects that are associated with a par- 

35 ticular piece of data. Buttons are objects that are 
associated with a particular function or action; when 
a user selects a button, the user interface performs 
the function or action associated with the button. 
Portals are objects that are displayed in a first 

40 Space but are associated with a second Space; 
when a user selects a portal the user passes 
through the portal and moves to the second Space 
associated with that portal. The user environment is 
further enhanced by the use of sound, coordinated 

45 with every user action and supplementing the vi- 
sual feedback provided. 

To control real world remote devices, many of 
the graphic objects presented on the screen are 
associated with remote devices in the real world. 

so To control a remote device, a user selects the 
graphic object pictured on the display screen asso- 
ciated with the particular remote device. After the 
user selects a graphic object associated with a 
remote device, the hand-held display device dis- 
ss plays a user interface for controlling the remote 
device. The user can then interact with the remote 
device's user interface. To control the remote de- 
vice, the present invention invokes a device driver 
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method for the remote device. The device driver 
method may be located within the remote device or 
within the local hand-held display device, depend- 
ing on the remote device's sophistication. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 

The objects, features and advantages of the 
present invention will be apparent to one skilled in 
the art, in view of the following detailed description 10 
in which: 

FIGURE 1a illustrates an overview of the remote 
control functionality provided by the present in- 
vention. 

FIGURE 1b illustrates an overview of the hard- 15 
ware used to implement the hand-held display 
device of the present invention. 
FIGURE 1c illustrates a block diagram of an 
intelligent remote device that is controlled by 
the present invention. 20 
FIGURE 1d illustrates a block diagram of a 
simple remote device that is controlled by the 
present invention. 

FIGURE 1e illustrates a block diagram of con- 
ventional electronic device that is controlled re- 25 
motely by the present invention. 
FIGURE 2a illustrates a structure of different 
geographic spaces. 

FIGURES 2b - 2c illustrate a sample geographic 
map structure of spaces used to provide a user 30 
interface. 

FIGURE 3 illustrates a view of the outer-pace 
Space of the user interface based upon the 
geographic map structure in Figures 2b and 2c. 
FIGURE 4 illustrates a first view of the Earth's 35 
surface Space in the user interface based upon 
the geographic map structure in Figures 2b and 
2c. 

FIGURE 5 illustrates a second view of the 
EartlYs surface Space in the user interface 40 
based upon the geographic map structure in 
Figures 2b and 2c. 

FIGURE 6 illustrates a view of the first house 
Space of the user interface based upon the 
geographic map structure in Figures 2b and 2c. 45 
FIGURE 7 illustrates a view of a living room 
Space in the user interface based upon the 
geographic map structure in Figures 2b and 2c. 
FIGURE 8 illustrates a view of a television Space 
in the user interface based upon the geographic 50 
map structure in Figures 2b and 2c. 
FIGURE 9 illustrates a view of a television pro- 
gram listing book Space in the user interface 
based upon the geographic map structure in 
Figures 2b and 2c. 55 
FIGURE 10 illustrates a view of a single televi- 
sion program description Space of the user in- 
terface based upon the geographic map struc- 



ture in Figures 2b and 2c. 

FIGURE 11 illustrates a conceptual view of the 

television program listing book Space of Figure 

9. 

FIGURE 12 illustrates a view of a plurality of 
objects which do not fit on a single screen 
display. 

FIGURE 13 illustrates a conceptual view of the 
plurality of objects placed on an object wheel. 
FIGURE 14 illustrates a conceptual view of the 
object wheel of Figure 13 where the object 
wheel has rotated revealing new objects. 
FIGURE 15 illustrates the Spin gesture being 
used to spin an object wheel. 
FIGURE 16a illustrates a Pickup gesture being 
used to pickup an object off an object wheel. 
FIGURE 16b illustrates a Pickup gesture being 
used to pickup an object from a Space. 
FIGURE 17a illustrates the Agent Portal of the 
present invention wearing a doctor's uniform to 
symbolize a diagnostic mode. 
FIGURE 17b illustrates the Agent Portal of the 
present invention holding a pencil to symbolize 
a writing mode. 

FIGURE 17c illustrates the Agent Portal of the 
present invention attempting to alert the user 
about a recent event. 

FIGURE 18a illustrates an intelligent device 
transmitting its graphical user interface to a dis- 
play device. 

FIGURE 18b illustrates a display device display- 
ing the graphical user interface of an intelligent 
controllable device. 

FIGURE 18c illustrates the display device trans- 
mitting user interactions with the user interface 
to the intelligent device. 

FIGURE 18d illustrates the intelligent device 
transmitting return values or feedback to the 
display device. 

FIGURE 19a illustrates a conceptual diagram of 
the program objects in the intelligent device 
being sent to the display device. 
FIGURE 19b illustrates a conceptual diagram of 
the program objects of the intelligent device in 
the display device. 

FIGURE 19c illustrates a conceptual diagram of 
the display device invoking a method in an 
object within the remote intelligent device. 
FIGURE 19d illustrates a conceptual diagram of 
the method in an object within the remote intel- 
ligent device returning status. 
FIGURE 20 illustrates a flowchart of the transfer 
of an intelligent device's user interface. 
FIGURE 21a illustrates a state diagram of the 
RPC manager for a client requesting a remote 
method invocation. 

FIGURE 21b illustrates a state diagram of the 
RPC manager for a server handling a remote 
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method invocation. 

FIGURE 21c illustrates the sequence of a re- 
mote method invocation. 

FIGURE 22a illustrates the simple device trans- 
ferring its graphical user interface to a display 
device. 

FIGURE 22b illustrates a display device display- 
ing the graphical user interface of the simple 
controllable device. 

FIGURE 22c illustrates the display device trans- 
mitting simple control codes to the simple de- 
vice. 

FIGURE 22d illustrates the simple device trans- 
mitting return values or feedback to the display 
device. 

FIGURE 23a illustrates a conceptual diagram of 
the program objects in a simple device being 
sent to the display device. 

FIGURE 23b illustrates a conceptual diagram of 
the program objects of the simple device in the 
display device. 

FIGURE 23c illustrates a conceptual diagram of 
the display device invoking a method in a pro- 
gram object and transmitting control codes to 
the simple device. 

FIGURE 23d illustrates a conceptual diagram of 
the remote simple device returning status in- 
formation. 

FIGURE 24 illustrates a flowchart of the transfer 
of a simple device's user interface. 

NOTATION AND NOMENCLATURE 

The detailed descriptions which follow are pre- 
sented largely in terms of display images, algo- 
rithms, and symbolic representations of operations 
of data bits within a computer memory. These 
algorithmic descriptions and representations are 
the means used by those skilled in the data pro- 
cessing arts to most effectively convey the sub- 
stance of their work to others skilled in the art. 

An algorithm is here, and generally, conceived 
to be a consistent sequence of steps leading to a 
desired result. These steps are those requiring 
physical manipulations of physical quantities. Usu- 
ally, though not necessarily, these quantities take 
the form of electrical or magnetic signals capable 
of being stored, transferred, combined, compared, 
selected, chosen, modified, and otherwise manipu- 
lated. It proves convenient at times, principally for 
reasons of common usage, to refer to these signals 
as bits, values, elements, symbols, characters, im- 
ages, terms, numbers, or the like. It should be 
borne in mind, however, that all of these, as well as 
similar terms, are to be associated with the appro- 
priate physical quantities and are merely conve- 
nient labels applied to these quantities. 



In the present case, the operations are ma- 
chine operations performed in conjunction with a 
human operator. Useful machines for performing 
the operations of the present invention include gen- 

5 eral purpose digital computers or other similar de- 
vices. In all cases, the distinction between the 
method operations of operating a computer and the 
method of computation itself should be kept in 
mind. The present invention relates to method 

70 steps for operating a computer and processing 
electrical or other physical signals to generate oth- 
er desired physical signals. 

The present invention also relates to apparatus 
for performing these operations. This apparatus 

;s may be specially constructed for the required pur- 
poses or it may comprise a general purpose com- 
puter selectively activated or reconfigured by a 
computer program stored in the computer. The 
algorithms presented herein are not inherently re- 

20 lated to any particular computer or other apparatus. 
In particular, various general purpose machines 
may be used with programs in accordance with the 
teachings herein, or it may prove more convenient 
to construct more specialized apparatus to perform 

25 the required method steps. The required structure 
for a variety of these machines will appear from the 
description given below. Machines which may per- 
form the functions of the present invention include 
those manufactured by the Assignee, FirstPerson, 

30 Inc., as well as other manufacturers of computer 
systems, including Sun Microsystems, Inc. of 
Mountain View, California. 

COOING DETAILS 

35 

No particular programming language has been 
indicated for carrying out the various procedures 
described herein. This is due in part to the fact that 
not all languages that might be mentioned are 

40 universally available. Each user of a particular com- 
puter will be aware of a language which is most 
suitable for his immediate purposes. In practice, it 
has proven useful to substantially implement the 
present invention in a high level language which is 

45 then compiled into machine executable object 
code. Because the computers and the monitor sys- 
tems which may be used in practicing the instant 
invention consist of many diverse elements, no 
detailed program listing has been provided. It is 

so considered that the operations and other proce- 
dures described herein and illustrated in the ac- 
companying drawings are sufficiently disclosed to 
permit one of ordinary skill to practice the instant 
invention. 

55 
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DETAILED DESCRIPTION OF THE INVENTION 

A computer based remote control system with 
an intuitive graphical user interface is disclosed. In 
the following description, for purposes of explana- 
tion, specific nomenclature is set forth to provide a 
thorough understanding of the present invention. 
However, it will be apparent to one skilled in the art 
that these specific details are not required to prac- 
tice the present invention. In other instances, well 
known circuits, functions, processes, and devices 
are shown in block diagram and conceptual dia- 
gram form in order not to obscure the present 
invention unnecessarily. 

System Overview 

Figure 1a illustrates an overview of the com- 
puter based remote control system of the present 
invention. The main component of the present in- 
vention consists of a hand-held display device 170 
as illustrated in Figure 1a. In the present embodi- 
ment, the hand-held display device 170 comprises 
a computer system with a color graphic touch 
screen display. The hand-held display device 170 
is designed to control any compatible remote de- 
vice such as thermostat 150, video cassette re- 
corder 39, and stereo system 152. The hand-held 
display device 170 communicates with the remote 
devices using a communications medium 160. 

The remote control system of the present in- 
vention as illustrated in Figure 1a differs signifi- 
cantly from prior art remote control systems since 
it is based upon modular object-oriented software. 
The object-oriented software provides features pre- 
viously not available in prior art remote control 
systems. For example, when an operator (some- 
time referred to herein as a "user") adds a new 
compatible remote device, such as thermostat 150 
to the system, the new remote device transmits 
information about itself to the hand-held dispjay, 
device 170. The_transmitted information includes a 
user interface object that can be used to render^a 
graphical user interface for the remote device. The 
present invention uses the user interface object to 
display a graphical User interface for "the new~re- 
mote device on. J&ejnand-held display device 170. 
The transmitted information may also include a 
device driver object that is used to control the ne.w 
remote device. To organize all the available user 
interface objects, the hand-held display device 170 
creates a "world" of geographic Spaces that can 
be easily be navigated using a set of gestures. 

This Specification will first disclose the hard- 
ware required to construct the hand-held display 
device 170 and the compatible remote devices of 
the present invention. The graphical user interface 
comprising geographic Spaces and objects which 



is displayed on the hand-held display device 170 
will then be described. A set of gestures used by 
operators to navigate around the user interface of 
the present invention will also be defined. To com- 

5 bine the user interface elements, a tour through an 
embodiment of the user interface of the present 
invention will be provided to illustrate how the 
navigational gestures are used to interact with the 
objects displayed on the hand-held display device 

70 170. Finally, the system of obtaining information 
from the remote devices is defined, such that the 
hand-held display device 170 may communicate 
with and control remote external devices. 

Due to the limitations of a written Specification 

75 to describe animated events on an interface, the 
reader is referred to a video tape entitled "Green 
Project User Interface," submitted by the applicant 
concurrently with the filing of the application on 
which this patent is based. In addition it must also 

20. be noted that the features of the present invention 
are illustrated in black and white within the accom- 
panying figures. However, in the presently pre- 
ferred embodiment, and as described below, ob- 
jects and features are displayed in color. 

25 

Hardware Configuration 

The hand-held display device 170 of the 
present invention comprises a computer system, 

30 such as that illustrated in Figure 1b. The computer 
system 170 of Figure 1b includes a central pro- 
cessing unit (CPU) 30, a memory unit 33, and an 
input/output (I/O) unit 34. These elements are those 
typically found in most general purpose computers, 

35 and in fact, the hand-held computer system 170 is. 
intended to be representative of a broad category 
of data processing devices capable of generating 
graphic displays. 

Coupled to the input/output (I/O) unit 34 of 

40 computer system 170 is a color graphic screen 
display 37. The screen display 37 comprises, in 
part, a color raster display device such as a cath- 
ode ray tube (CRT) monitor or a liquid crystal 
display (LCD). The screen display 37 must be of 

45 sufficient resolution such that screen display 37 
can render graphic images. 

In the preferred embodiment, the screen dis- 
play 37 further comprises a touch screen display 
system. The touch screen display includes a fea- 

50 ture that can detect the presence of a finger touch- 
ing the display screen 37. Furthermore, the touch 
screen can identify the precise location of the 
finger touching the display screen 37 such that the 
display screen can sense finger gestures made by 

55 a user's finger on the display screen 37. The touch 
screen display 37 may comprise one of a variety of 
touch sensitive display screens commercially avail- 
able on the market. 
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Alternatively, the hand-held computer system 
170 may be constructed with a conventional graph- 
ic display system without a touch sensitive display 
screen. Such a system requires an alternative input 
system to obtain the user's interactions. Figure 1b 
illustrates an alternative input device 42. Although 
the user interface of the present invention may be 
used with alternative input devices 42 such as 
mice, trackballs, and joysticks, the system will be 
described within this specification as the preferred 
embodiment with a touch screen display 37. 

In operation, the CPU 30 generates and dis- 
plays a graphical user interface on touch screen 
display 37 through the execution of computer code 
stored in memory 33. The computer code stored in 
memory 33 is not disclosed herein since the de- 
scription of the present invention in this Specifica- 
tion is sufficient for one skilled in the computer art 
to utilize the teachings of the invention in a variety 
of computer systems using one of many computer 
languages. 

Coupled to the input/output (I/O) unit 34 of 
computer system 170, in Figure 1b, is an external 
communication circuit 35. The external commu- 
nication circuit 35 provides a system for transmit- 
ting and receiving information to and from the 
hand-held computer system 170. The present in- 
vention uses the external communication circuit 35 
in computer system 170 to communicate with re- 
mote devices such as a video cassette recorder 39 
and a thermostat 150, across communications me- 
dium 160. The communication circuit 35 may use 
one of many well-known communication technol- 
ogies such as radio waves, low power, cellular, 
infrared signals, or data signals carried on existing 
power lines. 

To be controlled by the present hand-held dis- 
play device 170 of the present invention, a remote 
device must have at least one method for receiving 
information transmitted by the hand-held display 
device 170. Ideally, the remote devices are intel- 
ligent devices that can both transmit and receive 
information. However, the hand-held display device 
170 can control any remote device that fits within 
any of the three following remote device "types". 

The first type of remote device that can be 
controlled by the present invention is an "intelligent 
remote device" specifically designed to operate 
with the hand-held display device 170. In the 
present invention, an "intelligent remote device" is 
defined as a remote device that contains a micro- 
processor, a communication system, and a mem- 
ory. Figure 1c provides a block diagram of the 
components in an intelligent remote device 150. As 
illustrated in Figure 1c, the intelligent remote de- 
vice 150 comprises a CPU 403, a memory 404, a 
communications circuit 401, and a device control 
circuitry 405. 



The hand-held display device 170 of the 
present invention communicates with an intelligent 
remote device 150 through the intelligent remote 
device's communications circuit 401. The memory 

5 404 stores programs and data used by the CPU 
403, including,^ user interface object 303 and a 
device driver 7 object 301. The CPU 403 in the 
intelligent remote device invokes methods in the 
device driver object 301 to control the remote 

10 device. 

The second type of remote device that can be 
controlled by the present invention is a "simple 
remote device" specifically designed to operate 
with the hand-held display device 170. In the 

75 present invention, a remote device is defined as a 
"simple remote device" if the remote device does 
not have a microprocessor or the remote device 
contains such a primitive processor that the hand- 
held display device 170 can not invoke methods in 

20 a device driver object within the remote device. 

Figure 1d provides a block diagram of the 
components in a typical simple remote device 155. 
As illustrated in Figure 1d, the simple remote 
device 155 comprises an emitter 420, a receiver 

25 421, a memory 424, and device control circuitry 
425. The emitter 420 in the simple remote device 
155 is used to transmit the contents of the memory 
424 to the hand-held display device 170. There- 
fore, the simple remote device 155 transmits both 

30 the user interface object 353 and the device driver 
object 351 in the memory 424, such that the hand- 
held display device receives both objects. 

To control the simple remote device 155, the 
display device 170 invokes a method within the 

35 device driver object 351 to translate the user's 
interactions into simple predefined control codes. 
The display device 170 then transmits the simple 
predefined control codes to the simple remote de- 
vice 155 which receives the control codes though 

40 receiver 421 and passes the information to the 
device control circuitry 425. 

Finally, the third type of remote device that can 
be controlled by the present invention is a conven- 
tional electronic device that can be controlled by 

45 transmitted signals such as infrared signals. There- 
fore, the present invention is backward compatible 
with most existing devices that can be controlled 
remotely. Figure 1e illustrates a typical electronic 
device 157 that can be controlled by coded sig- 

so nals. The device has a receiver 450 for receiving 
transmitted coded control signals and device con- 
trol circuitry 455 for responding to the coded con- 
trol signals. The hand-held display device 170 con- 
trols the electronic device 157, by transmitting cod- 

55 ed control signals. The remote electronic device 
157 receives the transmitted signals and responds 
accordingly. 
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The Display Device User Interface 

In order to create a new product that is easy to 
use, product designers attempt to make the new 
product operate in a manner similar to an old 
product that the intended operator already knows 
how to operate. For example, audio cassette tape 
recorders have been available for such a long time 
that most people know how to operate an audio 
cassette tape recorder. The play, stop, rewind, fast- 
forward, and record buttons on cassette tape re- 
corders are therefore familiar to most people. Video 
cassette recorder manufacturers produced easy-to- 
use video cassette recorders by taking advantage 
of this common knowledge. The video cassette 
recorders are easy to use since the video cassette 
recorders used the same buttons, i.e., the play, 
stop, rewind, fast-forward, and record buttons, that 
audio cassette recorders utilize for operation. 

To make the hand-held display device 170 
easy to use, the present invention displays an 
intuitive graphical user interface on the hand-held 
display device 170. This section defines the graphi- 
cal user interface which is provided to users on the 
hand-held display device 170. 

To design a graphical user interface that is 
easy to, navigate, designers should organize the 
graphical user interface with a structure that most 
users already understand. Humans live in a three 
dimensional geographic world. To survive in our 
three dimensional geographic world people must 
learn how to navigate around the local geographic 
area, including such geographic spaces their 
house, their neighborhood, and their town. In order 
to do this, each person creates a mental map of 
geographic spaces of their relevant local world. 

The mental maps of geographic spaces people 
create are de-facto geographic map structures. 
Figure 2a illustrates a typical geographic map 
structure. For example, every person on this planet 
knows that they live on the Earth. Each person also 
lives in a certain town; yet, each person is aware 
that other towns exist on the Earth. Each person 
lives at a particular house on a particular street; 
yet, each person is aware that there are other 
streets in the town with other houses on each 
street. The identification of one particular house on 
one street within one town on Earth is a naturally 
occurring geographic map structure as illustrated in 
Figure 2a. 

The geographic map structure can be ex- 
tended into the house. For example, the living 
room is one room of many within the house. The 
television, television program schedule and the vid- 
eo cassette recorder are objects within the living 
room. Each object can have its own geographic 
space. The map structure can also be extended to 
abstract spaces as well as geographic spaces. 



Each television program listed in a television pro- 
gram schedule has a program description. The 
program description can be considered a space 
within the television program schedule. Most peo- 

5 pie understand both the structure of geographic 
spaces and the detail that will be found within each 
geographic space. Therefore the de-facto geo- 
graphic map structure of spaces represents a 
structure that is familiar to most people. 

10 The present invention uses the familiar geo- 

graphic map structure of our three dimensional 
world in a graphical user interface. The present 
invention describes a technique of user navigation 
in an arbitrarily large and complex geographic map 

75 structure through graphic background images that 
depict geographic spaces. Such geographic map 
structures improve a user's memory of complex 
data organization. The advantage of including si .h 
geographic map structures is that the familiar strt .- 

20 tures enable a wider range of people to use in- 
formation storage facilitated by a computer. 

Figures 2b and 2c illustrate a sample geo- 
graphic map structure utilized by the present in- 
vention. In the present invention, each Space in the 

25 geographic map structures of Figures 2b and 2c 
is represented as a background image on the dis- 
play screen of hand-held display device such as 
the one illustrated in Figure 1b. Located within 
each geographic Space are objects that can be 

30 selected and manipulated by the user. 

User Interface Display Ele ments 

The user interface environment of the present 

35 invention consists of two main display elements: 
"Spaces" and selectable "objects." Each area of 
the geographic map structure of Figures 2b and 
2c represents a Space. The Space provides a 
geographic reference that informs a user of the 

40 user's current location within the geographic map 
structure. Within each Space are one or more 
"objects." A user may select and manipulate the 
objects within each Space. In the present embodi- 
ment of the hand-held display device 170 with 

45 touch screen display 37, a user may select objects 
by touching the objects displayed on the touch 
screen display 37. Pointing devices other than the 
users finger and input devices, other than a touch 
screen, might handle selection and user input in a 

50 different manner. 

Spaces 

Each Space in Figures 2b and 2c has an 

55 associated background image that contains appro- 
priate imagery for the Space. When a user enters a 
Space, the Space's background image is rendered 
on the display screen 37. Each background image 

8 
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contains landmarks that help the user identify 
where the user is. For example, the Earth's surface 
Space 32 of Figure 2b is illustrated as the back- 
ground images in Figure 4 and Figure 5. The 
Earth's surface background image of Figures 4 
and 5 resembles a street on the Earth's surface, 
thereby informing a user that the user is at the 
Earth's surface Space of the map structure. The 
system renders the background image using de- 
saturated or dimmed colors such that the back- 
ground image can easily be distinguished from 
Selectable objects. The background image for a 
Space may be larger than the display screen 37; 
therefore, the present invention provides a gesture 
for "panning" such that a user may move around 
within a Space as will be explained later. 

Objects 

Objects are individual graphical icons that are 
rendered within the background images of Spaces. 
Each object is actually an object-oriented program- 
ming type of object containing data methods which 
can be invoked to act on the data. A user can 
select and manipulate any of the objects which 
appear within the background image of a Space. 
Each object is brightly colored and outlined in 
black. The bright colors distinguish the objects in a 
Space from the Space's background image that is 
drawn using desaturated colors. When a user 
points to an object, the user interface generates a 
"Selectable Object" sound that informs the user 
that the object can be selected and manipulated. 
Referring again to Figure 4, in the present exam- 
ple, the houses 47 and 48 are objects that can be 
selected and manipulated by a user. 

An "anticipation animation" is used in the 
present invention to inform the user when some 
action is taking place. The anticipation animation is 
a short animation displayed on the display screen 
37 that is generated by the CPU 30. In most cases, 
the anticipation animation consists of enlarging an 
object's icon such that the icon appears to move 
closer to the user when the object is selected. 

For example, referring to Figure 3, when the 
Earth portal 100 is selected, the Earth portal 100 
expands such that it appears to move closer to the 
user. After the Earth portal 100 anticipates, the user 
interfaces move the user to the Earth's surface 
Space 32 that is associated with the Earth portal 
100 as illustrated in Figure 4. 

When a user selects an object by pointing to 
the object for a predetermined minimum period of 
time, a sound is generated, the object anticipates, 
and the CPU invokes a method within the object 
which performs an action associated with the ob- 
ject. 



There are at least three types of objects that 
exist in the user interface of the present embodi- 
ment: data objects, buttons, and portals. The sim- 
plest object is a data object. Data objects are 
5 objects associated with a set of data. Buttons are 
objects that provide a means for accomplishing 
functions. Portals provide a method for moving 
between different Spaces. Each type of object will 
be explained individually below. 

70 

Data Objects 

A data object is a simple object used to repre- 
sent some piece of information. For example, a 

;s data object may represent a color or a television 
channel. The data objects have no methods that 
can be invoked to perform an associated action or 
function when the data object is selected. However, 
a data object can be dropped onto other objects 

20 that use the information in the data object. For 
example, a television program listing data object 
can be dropped onto a video cassette recorder 
object to program the video cassette recorder to 
record the television program. 

25 

Buttons 

A button object is a special object that has an 
associated action or function. When a user selects 

30 a button, the CPU 30 invokes the action method in 
the button object to cause the execution of the 
action or function associated the button. When the 
button is selected, the CPU also anticipates the 
button and generates a "button action" sound pro- 

35 viding feedback to the user that the button has 
been selected. Certain buttons provide additional 
visual feedback to inform the user of the results of 
the method invoked by the CPU to perform the 
action associated with the button. 

40 For example, referring to Figure 8, a television 

power "on" button 61 and a television power "off" 
button 62 are displayed. When a user selects the 
television power "on" button 61, the television pow- 
er "on" button 61 anticipates and the CPU 30 

45 generates the "button action" sound. The CPU 30 
also invokes a method in the button object that 
transmits the necessary information to turn on a 
real television set associated with the television 
control panel illustrated in Figure 8. 

so If the real television set is an "intelligent re- 

mote device", the television set may return a result 
value to the display device informing the user 
whether or not the television was turned on. The 
user interface may then light up the television 

55 power "on" button 61 and generate a "hum" sound 
to inform the user that the real television was 
actually turned on. 
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Buttons can be copied and moved to different 
Spaces so that a user can customize the "world" of 
the user interface. For example, the television pow- 
er "on" 61 and the television power "off" 62 but- 
tons could be copied, and the button copies moved 
from the television control Space to any other 
Space. A user may use the copies of the television 
power "on" 61 and the television power "off" 62 
buttons in the new Space to control the television. 

Portals 

Portals are objects that provide a simple meth- 
od of moving from the current Space to another 
Space associated with the portal. A user moves 
from the current Space to another Space asso- 
ciated with a portal by simply Selecting the portal. 
After a user selects a portal, the portal anticipates, 
generates a "movement through portal sound," and 
the CPU invokes a method that moves the user to 
the Space associated with the portal. 

For example, referring to the geographic map 
of Figure 2b, to move from the Earth's surface 
Space 32 to the house Space 50, a user simply 
selects the house portal 48. Referring to the Earth's 
surface Space 32 as illustrated in Figure 4, a user 
performs the move by selecting the house portal 
48 within the Earth's surface Space 32. After being 
selected, the house portal 48 anticipates, generates 
a movement through portal sound," and then the 
user is moved to the house Space 50 as illustrated 
in Figure 6. 

Portals can be duplicated and moved such that 
a user can customize the "world" of the user 
interface. For example, in Figure 7 a television 
program schedule portal 71 is illustrated within the 
living room Space 60. The TV program schedule 
portal 71 is associated with a Space that contains 
TV program schedule information. A user can make 
a copy of the television program schedule portal 71 
and move~ the copy to another Space where the 
user would like to be able to access the television 
program schedule. The user can then use the copy 
of the television program schedule portal to access 
the same information that could be accessed using 
the original television program schedule portal. 
Thus, a user would be able to access the same 
information from different Spaces. 

There are two special portals that are almost 
always available to an operator of the user inter- 
face: a Wayback portal and an Agent portal. 

The Wayback Portal 

The Wayback portal provides a way for a user 
to trace back a path that the user has traveled 
along between different Spaces. The Wayback por- 
tal 45 consists of a small rectangular frame that 



encloses a snapshot of the previous Space along 
the path of Spaces that the user traveled along. 
Since the background image for a Space can be 
larger than what can be seen in a single view, the 

5 Wayback portal shows a snapshot of the view of 
the previous Space exactly as it appeared when 
the user went through a portal in that previous 
Space. In the present embodiment, the Wayback 
portal is available to the user in every Space in the 

70 lower right-hand corner of the display screen. If 
desired, the user can move the Wayback portal to 
a new location on the display screen such that the 
Wayback portal will then appear at the new location 
on the screen in every Space. 

75 To move one step backward along the current 

path, the user Selects the Wayback portal 45. 
When Selected, the CPU 30 anticipates the 
Wayback portal 45, generates a "move through 
Wayback portal" sound, and moves the user to the 

20 Space pictured within the Wayback portal. In the 
present embodiment, the "move through Wayback 
portal" sound is actually the same sound as the 
"move through portal" sound except that it is 
played backwards. By playing the "move through 

25 portal" sound backward when moving through the 
Wayback portal, the movement backward along the 
current path is emphasized to the user. 

As users go through portals they leave a trail of 
invisible markers. Each marker represents a visit to 

30 a Space. The Wayback manages these markers 
and maintains them in the order that the Spaces 
are visited. The markers and the markers' order 
define the path that the user has taken through the 
world of the user interface. Thus, when a user goes 

35 though the Wayback, the path defined by the mark- 
ers is retraced one marker at a time. Each time the 
user goes through the Wayback to a Space, a 
marker is removed from those currently managed 
by the Wayback. 

40 For example, referring to geographic map 

structure illustrated in Figure 2b, if a user starts in 
the outer-space Space 99, moves to the Earth's 
surface Space 32, and then moves to the house 
Space 50; a path will be generated from the outer- 

45 space Space 99 to the Earth's surface Space 32 to 
the house Space 50. After traversing this path, the 
display screen 37 will display the house Space 50 
as depicted in Figure 6. In the lower right corner of 
the house Space 50 is the Wayback portal 45. As 

so illustrated in Figure 6, the Wayback portal 45 dis- 
plays a miniature snapshot of the previous Space 
along the Space path. In. this case, the previous 
Space along the Space path is the Earth's surface 
Space 32; therefore, the Wayback portal 45 dis- 
ss plays a miniature snapshot of the Earth's surface 
Space 32 as it appeared when the user went 
through the house portal. If the user then Selects 
the Wayback portal 45, the Wayback portal 45 
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anticipates, generates the "move through Wayback 
portal" sound, and then moves the user to the 
Earth's surface Space 32 as depicted in Figure 4. 
At the Earth's surface Space 32 as depicted in 
Figure 4 the Wayback now displays a miniature 
snapshot of the outer-space Space 99, which was 
the first Space along the path. The Wayback portal 
45 therefore provides a "way back" along the path 
of Spaces the user has traveled. 

The present invention implements the 
Wayback portal 45 as a stack of data structures, 
wherein each data structure contains a pointer and 
a pair of coordinates. The pointer in each data 
structure on the stack points to a Space that the 
user has previously visited. The pair of coordinates 
in each data structure on the stack specifies ex- 
actly where the user was within the Space pointed 
to by the pointer when the user left the Space 
through a portal. Each time the user moves through 
a portal from a particular Space, the user interface 
"pushes" a data structure, containing the pointer to 
the Space and the coordinate pair which defines 
exactly where the user was within the Space when 
he went through the portal, onto the wayback stack. 
The Wayback portal 45 displays a miniature snap- 
shot of the Space that is pointed to by the pointer 
in the data structure on the top of the stack. To 
trace back along the path of Spaces, each time the 
user Selects the Wayback portal 45, the user inter- 
face "pops" a data structure off the stack and 
moves to the Space indicated by the pointer in the 
data structure. 

The Agent Portal 

The Agent is an anthropomorphic character 
that aids the user in navigating around the user 
interface and in dealing with events that occur in 
the "world" created by the user interface. Referring 
to the Earth's surface Space of Figure 4, the Agent 
41 is illustrated in the lower left corner. The Agent 
41 carries objects around for the user. This carry- 
ing function is represented by a briefcase, which 
the Agent also carries. A user can access objects 
that the Agent is carrying by Selecting the Agent 
portal 41 thereby entering an Agent Space asso- 
ciated with the Agent portal 41. When a user Se- 
lects the Agent portal 41, the Agent anticipates by 
appearing to pull out the briefcase and open the 
briefcase up. The CPU then generates the "move 
through portal" sound and moves the user to the 
Agent Space. 

To place an object into the Agent's briefcase, a 
user picks up the object and then "Drops" the 
object onto the Agent portal 41. The object 
Dropped onto the Agent portal 41 is then always 
accessible to the user from any Space by entering 
the Agent's Space. The Agent is discussed more 



fully below in the section entitled "The Agent." 
Object Wheel 

5 It is often necessary that a user make choices 

among an large, arbitrary number of objects. 
Therefore, the user interface must provide both a 
method of displaying a large number of objects 
and a method of allowing the user to Select any of 

/o the displayed objects. 

For example, referring to the television control 
Space 64 illustrated in Figure 8, the user interface 
of the present invention must provide a method of 
displaying an arbitrary number of television chan- 

75 nels for the user to choose from. In any system, 
screen space is at a premium, so when the user 
must choose among many possible options, the 
options must be presented compactly. In the 
present invention, if there are 500 television chan- 

20 nel objects for a user to select from, presenting 
them all at once would not only take up significant 
screen space, but the visual context of the list (i.e., 
why is the user looking at this list in the first place) 
would be lost. Thus, a method for allowing the. user 

25 to browse through the available objects is required. 

The user interface of the present invention in- 
troduces an "object wheel" to provide a method for 
allowing a user to choose from a large group of 
objects. In the object wheel, a few immediately 

30 accessible objects are displayed in an object wheel 
on the display screen while the remaining objects 
in the group are not rendered on the screen. To 
access the objects not rendered on the screen, the 
user "Spins" the object wheel to move the pre- 

35 viously inaccessible objects into view on the dis- 
play screen. 

Referring to Figure 12, a large number of fruit 
objects are available for a user to choose from, yet 
only four fruit objects fit horizontally on the display 

40 screen 37 at once. To provide the user with access 
to all the fruit objects available, the present inven- 
tion puts the strip of fruit objects onto an "object 
wheel" 38 as displayed in the conceptual diagram 
of Figure 13. A few of the objects on the object 

45 wheel 38 are immediately accessible to a user on 
the display screen 37. To access the objects that 
are not displayed on the display screen 37, the 
user rotates the object wheel 38. As the object 
wheel 38 rotates, new objects are brought into view 

50 and other objects disappear from view as illustrated 
in Figure 14. An operator uses the navigational 
gestures Point, Spin, and Pickup to manipulate the 
object wheel presented on the touch screen display 
37. These three navigational gestures are explained 

55 in the following section. 

In order to enhance the user's feeling that a 
real wheel is being spun, object wheels obey the 
laws of physics. For example, the object wheels 
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have mass and therefore inertia; object wheels spin 
faster as the user applies more "force" when spin- 
ning the wheel; object wheels respond to friction 
since a spinning object wheel gradually stops spin- 
ning. 5 

While an object wheel is spinning and the 
objects are moving across the screen, the current 
embodiment of the user interface generates a "clic- 
king" sound. The "clicking" sound generated by 
the user interface simulates the sound made by w 
detents on a slot machine wheel which forces the 
wheel to stop and be aligned. Like a real wheel, the 
object wheels of the present invention always stop 
aligned, and never stop in between object choices. 
A user can stop a spinning object wheel at any 15 
time by Pointing to a moving object on the object 
wheel while the object wheel is spinning. - 

An object wheel can be aligned vertically or 
horizontally on the display screen. When placed 
vertically, the object wheel can be spun both up 20 
and down. When placed horizontally, the object 
wheel can be spun both left and right. 

User Interface Navigational Gestures 

25 

Navigation and object manipulation in the user 
interface are performed using a screen pointing 
device. In the preferred embodiment, the hand-held 
display device 170 is constructed with a touch 
screen display 37 and the pointing device used is a 30 
user's finger. Alternatively, the user interface may 
be implemented using any other type of screen 
pointing device such as a mouse, a trackball, or a 
joystick. This specification, however, will only de- 
scribe the preferred embodiment that uses the 35 
user's finger as a pointing device on a touch 
screen. 

In the preferred embodiment of the present 
invention, the hand-held display device 170 com- 
prises, in part, a touch screen display 37 as illus- 40 
trated in Figure 1b. The touch screen display 37 
senses when a finger has been placed on the 
display screen and identifies where the finger has 
been placed. Users select objects on the touch 
screen and manipulate the objects using a set of 45 
six navigational gestures: Point, Select, Pan, Spin, 
Pickup, and Drop. Each navigational gesture used 
by the user interface is explained individually be- 
low. (Since the navigational gesture names are the 
same as common words, this specification will 50 
capitalize all references to the navigational ges- 
tures.) 

Point 

55 

Point is the most simple gesture in the user 
interface of the present invention. An operator uses 
the Point gesture to pick objects rendered on the 



display screen. For example, if the user interface of -J 
the present invention requests the user to choose j 
one of many objects rendered on the display \ 
screen, the user Points to an object to choose the j 
object. j 
A user Points to an object by simply placing a j 
finger on the object displayed on the touch screen. I 
When a user has successfully Pointed to an object, j 
the user interface generates a "Selectable Object" 
sound. The Selectable object sound provides feed- ! 
back to the user indicating that the user has sue- I 
cessfully Pointed to the object. 

Select ; 

-i 

The Select gesture is very similar to the Point i 
gesture. An operator uses the Select gesture to < 
have an object invoke a method to perform an ; 
function or action associated with the object. For ; 
example, a Selected portal invokes a method that 
moves the user to the Space associated with the 
portal and a Selected button invokes a method that =j 
performs the action or function associated with the I 
button. ] 

To Select an object, a user first Points to an ! 
object by touching the object with a finger. If the S 
user continues to Point to the object for a predeter- \ 
mined minimum period of time, the object be- 
comes Selected. After a user Selects an object, the j 
object "anticipates," generates a sound for user ; 
feedback, and then a method within the object is j 
invoked. If the object is a button, then the method 
invoked performs a function associated with the ? 
button. If the object is a portal, then the method :f 
invoked will move the user to the Space associated j 
with that portal. 

Alternatively, a user can Select an object on 
the display screen by quickly Pointing to the object 
twice in succession. When a user Selects an object 
by quickly Pointing to the object twice, the object 
does not anticipate. Instead, the user interface only 
generates the appropriate feedback sound and then 
performs the action associated with the object. This 
provides the user a method of Selecting an object 
quickly. 

Pan 

To move around within a Space a user uses 
the Pan navigational gesture. A user Pans though a 
Space by placing a finger on the background im- 
age of a Space displayed on the touch display 
screen 37 and sliding the finger in the desired 
direction of movement. The user interface of the 
present invention responds by generating a "Pan 
Active" sound and moving background image of 
the Space and all the objects within the Space in 
the opposite direction of the finger's movement. By 
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moving the Space in the opposite direction, the 
user interface simulates movement of the users' 
point of view in the other direction. 

Figures 4 and 5 illustrate the use of the Pan 
gesture. If a user places his finger on the back- 
ground of Figure 4 and then slides the finger on 
the touch screen to the right, the view will cor- 
respondingly pan to the right by moving the back- 
ground image of the Space and all the objects 
within the Space to the left. Figure 5 illustrates the 
result of the Pan right operation where house 48 
has moved from the right side of the display 
screen over to the left side of the display screen. 

The background image for each Space in the 
present invention can be arbitrarily large. If the 
background image of a Space has finite size, the 
user interface of the present invention provides 
feedback that the furthest extent of the background 
image has been reached. When a user Pans in a 
direction all the way to the end of a background 
image, the Pan gesture will not move the user any 
further in that direction. To inform the user of this 
fact, the user interface generates an "end of ex- 
tent" sound instead of the "Pan Active" sound 
when a user attempts to Pan past the end of a 
background image. The "end of extent" sound 
sounds like a "thud" made when something bumps 
into a wall. The "end of extent" sound thereby 
informs the user that the user has hit the end of the 
background image and can go no further. 

Spin 

An operator uses the Spin gesture to browse 
through several objects placed on an object wheel. 
When a user Spins an object wheel, the visible 
objects on the object wheel move across the dis- 
play screen and disappear off one end of the 
screen and previously invisible objects move onto 
the display screen at the other end. 

A user Spins an object wheel by placing a 
finger on the object wheel and then moving the 
finger across the display screen in a direction 
parallel to the object wheel's movement. Figure 15 
illustrates the Spin gesture. In Figure 15, a user 
first places a finger on the object wheel (as shown 
by the asterisk 96). The user then slides the finger 
across the object wheel (as shown by the arrow 
97). This finger gesture causes the object wheel to 
Spin in the direction of the finger's movement. The 
faster the user slides the finger across the object 
wheel on the display, the faster the object wheel 
spins. Once the object wheel is spinning, the ob- 
jects on the object wheel appear at one end of an 
object wheel, move across the display screen, and 
then disappear off the other end of the object 
wheel similar to figures which appear on a slot 
machine wheel. 



Pickup 

The Pickup gesture is used to pickup an object 
located within a Space or on an object wheel such 

5 that a user can carry the object around the "world" 
defined by the user interface of the present inven- 
tion. In the present embodiment, different methods 
are used to Pickup an object from a Space or an 
object wheel. Each method will be explained in- 

w dividually below. However, it will be appreciated 
that the same gesture could be used to pickup 
objects in both situations. 

The gesture used to Pickup an object from an 
object wheel is illustrated in Figure 16a. Figure 

15 16a illustrates an object wheel 91 with a number of 
fruit objects to choose from. To Pickup one of the 
fruit objects disposed on the object wheel 91, a 
user first places a finger on the object (as des- 
ignated by the asterisk 92) and moves the finger in 

20 a direction perpendicular to the motion of the ob- 
ject wheel. The movement of the finger in a direc- 
tion perpendicular to the motion of the object wheel 
is illustrated by arrow 93. The perpendicular finger 
motion "tears" the object off the object wheel such 

25 that the user can move the object. When the object 
comes loose from the object wheel, the user inter- 
face generates a "tearing" sound that informs the 
user that the object has been removed from the 
object wheel. When an object has been picked up, 

30 the object casts a shadow onto the background 
image of the Space and moves the object along 
with the user's finger. 

The gesture used to Pickup an object from a 
Space is illustrated in Figure 16b. Figure 16b 

35 illustrates a Space with an object in the center.. To 
Pickup the object, a user circles the object with the 
finger as illustrated by arrow 203 and then touches 
the object as illustrated by asterisk 201. When the 
object comes loose from the Space, the user inter- 

40 face generates the tearing sound that informs the 
user that the object has been picked up. After an 
object has been picked up from a Space by a user, 
a ghost image of the object remains at the object's 
original location. The ghost image of the object is 

45 drawn in desaturated colors indicating to the user 
that the ghost object cannot be Selected. 

After a user has picked up an object, either 
from an object wheel or a Space, the user interface 
generates a "carrying object" sound that informs 

so the user that an object is being carried. The user 
can move the carried object within a Space, from 
Space to Space, or Drop the carried object onto 
another object. 

55 Drop 

The final gesture used to interact with the user 
interface of the present invention is Drop. After a 
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user has used the Pickup gesture to pick up an 
object from a Space or from an object wheel, the 
user must continue to keep the finger pressed 
against the screen to carry the object. The user 
moves the carried object by sliding the finger 
across the touch screen display system. After mov- 
ing the carried object to a desired location, the 
user then uses the Drop gesture to place the object 
at the desired location. To drop the object the user 
lifts the finger off the touch screen display. When 
the user Drops an object, the object falls into the 
background image of a Space as if pulled by 
gravity, and the object no longer casts a shadow. 

A user of the present invention uses the Drop 
gesture to perform a number of different functions. 
The Drop gesture can be used to copy an object 
by picking up the object and Dropping the object 
into a portal such that a copy of the object is 
created in the Space associated with the portal. 
The Drop gesture can be used to move an object 
to a new location by picking up the object and 
Dropping the object at the desired location. The 
Drop gesture can also be used to provide a data 
object containing information to another object 
which uses the information to perform a function. 
Each different use of the Drop gesture will be 
explained below. 

To make a copy of an object using the Drop 
gesture, a user must first Pickup the object and 
then Drop the object onto a portal. An object that 
has been picked up is carried over the portal by 
sliding the finger across the screen. Once the car- 
ried object is over the portal, the portal becomes 
highlighted such that it provides feedback to the 
user signaling that the carried object is over the 
portal. In the present embodiment, the portal emits 
a green glow. To Drop the carried object into the 
portal, the user lifts the finger away from the touch 
screen display. The object is then displayed such 
that it passes through the portal with an appropriate 
animation. During the animation, the CPU 30 gen- 
erates a "sucked through portal" sound effect. 
When the object has completely disappeared 
through the portal, the ghost image of the object 
drawn in desaturated colors in the present Space 
changes back into a brightly colored object. This 
provides visual feedback to the user that the origi- 
nal object remains in the present Space even 
though a copy of the object has been sent through 
the portal to another Space. 

Users can move objects either within a single 
Space or from one Space to another. To move an 
object within a single Space a user first uses the 
Pickup gesture to pick up an object, moves the 
object to the desired new location, and then Drops 
the object using the Drop gesture. After the object 
is Dropped, the ghost image of the object merges 
with the Dropped object at the new location to form 



a single brightly colored image of the object at the 
new location. This informs the user that the object 
has been moved to a new location and that a copy 
has not been made. 
5 Moving an object to another Space is accom- 

plished by carrying the object through a portal to 
another Space. To move an object to another 
Space, a user must first Pickup the object and then 
carry the object over a portal. Once the carried 
w object is ovqr the portal, the portal becomes high- 
lighted such that it provides feedback to the user 
signaling that the carried object is over the portal, 
and the portal emits a green glow. To move the 
carried object through the portal, the user holds the 
75 object over the portal for a predetermined mini- 
mum amount of time. The portal eventually antici- 
pates moving the user and the carried object to the 
Space associated with the portal. The ghost image 
of the carried object drawn in desaturated colors is 

20 also moved along, with the user, and the carried 
object to the destination Space. When the user 
Drops the carried object by lifting his finger off of 
the touch screen display system, the carried object 
and the ghost image merge into a single brightly 

25 colored image of the object. This provides feed- 
back to the user that the object has been moved to 
the Space and has not been copied. 

The Drop gesture can also be used to provide 
information to an object. A user provides informa- 

30 tion to an object by Dropping an object containing 
data (i.e. a data object) onto the object which will 
use that data. 

To provide information to an object, a user 
must first Pickup a data object that contains the 

35 necessary information. The user then carries the 
data object over the object which can utilize the 
information. When the carried data object is over 
the object that can utilize the information, the ob- 
ject that can utilize the information becomes high- 

40 lighted if the carried data object is of the correct 
class. By becoming highlighted, the object that can 
utilize the information provides feedback to the 
user that the carried data object is of the correct 
class. To Drop the data object into the object that 

45 can utilize the information, the user lifts his finger 
used to carry the data object away from the touch 
screen. The data object is then passed through and 
into the portal with appropriate animation and 
sound effects. When the data object has com- 

50 pletely disappeared into the receiving object which 
can use the information, the receiving object then 
performs the appropriate function on the data pro- 
vided. 

For example, the Drop gesture can be used to 
55 provide television program information to the video 
cassette recorder object which is associated with a 
physical video cassette recorder, such that the 
video cassette recorder programs itself to record 
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the television program. To program the video cas- 
sette recorder, a user first uses the Pickup gesture 
to pickup a data object containing television pro- 
gram information. The user then carries a television 
program data object over the video cassette re- 
corder object. The video cassette recorder object 
then modifies its appearance to indicate it is a 
"program video cassette recorder" mode. This in- 
forms the user that the video cassette recorder 
"understands" the information that is over it (i.e., 
the video cassette recorder will be programmed to 
record the television program if the user Drops the 
television program data object onto the video cas- 
sette recorder). 

When the user Drops the television program 
object into the video cassette recorder, the video 
cassette recorder object recognizes that the televi- 
sion program data object contains information it 
can use. The video cassette recorder then extracts 
the time and channel information from the televi- 
sion program data object such that the video cas- 
sette recorder object programs itself to record the 
television program. 

A Tour of the Present Embodiment in Opera- 
tion 

To best convey the look and feel of the user 
interface of the present invention, a "tour" through 
a sample environment is provided with reference to 
Figures 3-10. The sample environment explored 
in the tour is based upon the geographic map 
structure of Spaces diagrammed in Figures 2b 
and 2c. The tour through the sample environment 
described herein is not intended to limit the scope 
of the present invention, but instead to provide an 
example of one possible embodiment of the 
present invention. Due to the limitations of a written 
Specification in describing the present invention's 
animated user interface, the reader is encouraged 
to view a video tape entitled "Green Project User 
Interface," submitted by the applicant concurrently 
with the filing of the application on which this 
patent is based. 

Referring to Figure 3, upon first entering the 
user interface of the present invention, the user 
interface initially positions the user in an outer- 
space Space 99. This Space is illustrated in the 
geographic map structure of Spaces diagrammed 
in Figure 2b. Although, the user is initially placed 
in the outer-space Space 99, the user interface will 
later start the user at the last Space the user was 
in when the user quit using the hand-held display 
device. Referring again to Figure 3, at the outer- 
space Space 99, the user interface renders a 
brightly colored Earth portal 100 against a dark 
outer-space background image on the display 
screen. 



Soon after entering the user interface, an out- 
line of a hand 31 appears on the display screen to 
teach the user how to navigate around the user 
interface. The hand 31 that appears on the display 

5 screen is the hand of the Agent. The Agent's hand 
31 extends a finger to touch the screen and then 
slides the finger across the screen causing the 
Space and objects on the screen display to Pan 
and the user interface to emit the "Panning active" 

70 sound. Referring to Figure 3, the outline hand 31 
of the Agent is shown moving across the screen. 
By touching the display screen and moving a fin- 
ger across the display screen, the Agent's hand 31 
demonstrates to a new user how to use the Pan 

15 gesture to move around the view of a Space. 

After demonstrating how to use the Pan ges- 
ture to move around in a Space, the hand 31 of the 
Agent next demonstrates how to use the Point and 
Select gestures. The Agent demonstrates how to 

20 use the Point gesture to Point to an object by 
placing a finger on the Earth portal 100. After the 
Agent Points to the Earth portal 100, the user 
interface emits the "Selectable Object" sound to 
provide feedback which informs the user that the 

25 Earth portal 100 is a Selectable Object. The Agent 
then demonstrates how to use the Select gesture 
to Select an object by continuing to Point to the 
Earth portal 100, After the Agent's hand 31 Points 
to the Earth portal 100 for the predetermined mini- 

30 mum amount of time, the Earth portal 100 antici- 
pates and the CPU generates the "move through 
portal" sound. The Earth portal 100 anticipates by 
expanding rapidly such that the Earth portal 100 
appears to move closer to the user. The screen 

35 then fades and a new background image appears 
displaying the Earth's surface Space 32 associated 
with the Earth portal 100, as illustrated in Figure 4. 

At the Earth's surface Space 32, the Agent 
portal 41 appears to "jump" onto the touch screen 

40 display 37 and moves to the lower left corner of 
the display screen. The Agent 41 will always be 
accessible to the user because the agent appears 
in every Space, except the Agent's Space. If de- 
sired, the user can move the Agent from the lower 

45 left corner to a different location on the display 
screen. This movement of the Agent will change 
the Agent's location in every Space. 

Referring to Figure 4, in the lower right corner 
of the display is the Wayback portal 45. The 

so Wayback portal 45 displays a framed snapshot of 
the outer-space Space 99 from Figure 3 which the 
user just came from. If the user Selects the 
Wayback portal 45, the user interface of the 
present invention moves the user back along the 

55 traveled path to the Space illustrated within the 
Wayback portal 45. For example, if the user de- 
sires to return to the outer-space Space 99 dis- 
played in the Wayback portal 45, the user simply 
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Selects the Wayback portal 45. 

The Earth's surface Space 32 depicted in Fig- 
ure 4 is comprised of a number of houses and 
community buildings such as stores, banks, 
schools, etc. Each building in the Earth's surface 
Space 32 represents a portal that can be Selected 
by the user to move the user to a Space asso- 
ciated with the portal. For example, in the Earth's 
surface Space depicted in Figure 4, the first house 
47 and the second house 48 are portals which can 
be Selected by the user. If the user Selects either 
the first house 47 or the second house 48, the user 
will be moved to an associated house Space. 

While within the Earth's surface Space de- 
picted in Figure 4, the user can move around in 
the Space by using the Pan gesture on the touch 
screen as demonstrated by the Agent's hand 31 in 
Figure 3. To Pan, the user simply places a finger 
on the screen and moves the finger in the direction 
the user wishes to move. The user interface re- 
sponds to the user's finger motion by generating 
the "Panning Active" sound and moving the back- 
ground image and the objects on the screen in the 
opposite direction of the finger's movement, there- 
by bringing other objects into the view. 

For example, if a user viewing the Earth's 
surface Space 32 as depicted in Figure 4 wishes 
to move to the right, the user simply places a 
finger onto a background area of the Earth's sur- 
face Space 32 and slides the finger to the right. 
The user interface reciprocally responds to the 
user's finger motion by moving the Space's back- 
ground image and the objects in the Space to the 
left. Figure 5 depicts the view of the Earth's sur- 
face Space 32 after the user has placed a finger 
onto a background area of Figure 4 and slid the 
finger to the right. A first house portal 47 displayed 
in Figure 4 has moved out of the view and is not 
visible in Figure 5. A second house portal 48 
displayed on the right side of the view in Figure 4 
has moved to the left side of the view in Figure 5 
and now is only partially visible. A bank portal 49 
that was not visible in Figure 4 has moved into 
view from the right as illustrated in Figure 5. The 
Agent 41 and Wayback portal 45 remain in the 
view in the same lower left and lower right posi- 
tions respectively. 

When the user Selects any portal from the 
Earth's surface Space 32 as depicted in Figures 4 
and 5, the Selected portal anticipates to provide 
feedback to the user and the user interface moves 
the user to a Space associated with the Selected 
portal. For example, when a user at the Earth's 
surface Space 32 depicted in Figure 4 Selects the 
first house portal 47, the house portal 47 antici- 
pates and then the user interface moves the user 
to a house Space associated with the house portal 
47. Figure 6 illustrates a house Space 50 asso- 



ciated with the house portal 47 of Figure 4. 

The house Space 50 displayed in Figure 6 
consists of a house with the outer walls removed 
such that the individual rooms inside the house are 
visible. All of the rooms inside the house, such as 
the living room 54, bedroom 53, bathroom 55, attic 
52, and kitchen 56 depicted in Figure 6, are por- 
tals that can be Selected by the user. As illustrated 
in Figure 6, the Wayback portal 45 contains a 
miniature snapshot of the Earth's surface Space 
32, as it appeared when the user left the previous 
Space. Thus the Wayback portal 45 depicted in 
Figure 6 not only contains a miniature view of the 
Earth's surface Space, but a view of that portion of 
the Earth's surface Space 32 that was in view when 
the user Selected the house portal 47 of Figure 4. 
To move back up to the Earth's surface Space of 
Figure 4, the user Selects the Wayback portal 45 
that contains a snapshot image of the Earth's sur- 
face Space. 

Referring to Figure 6, if the user Selects the 
living room portal 54, the living room portal 54 
anticipates, the "move through portal" sound is 
generated, and the user interface moves the user 
to a living room Space associated with the living 
room portal 54. Referring to Figure 7, an embodi- 
ment of a living room Space 60 is depicted. Within 
the living room Space 60 of Figure 7 is a television 
73, a video cassette recorder 75, and a television 
program schedule book 71. The television 73, vid- 
eo cassette recorder 75, and television program 
schedule book 71 are each portals that can be 
Selected by the user. The television 73 and the 
video cassette recorder 75 portals are associated 
with a real television and video cassette recorder 
respectively. 

When a user Selects the television portal 73 
displayed within the living room Space 60 of Fig- 
ure 7, the television portal 73 anticipates and the 
user interface displays a graphical user interface 
associated with the real television. Figure 8 depicts 
the television control Space 64 which comprises 
the television's graphical user interface in the 
present embodiment. The objects available in the 
television Space 64 include a green "on" button 
61, a red "off" button 62, and several television 
channel buttons 66 - 69. The television channel 
buttons 66 - 69 are on a horizontally aligned object 
wheel 63 such that a user can Spin the object 
wheel 63 left or right in order to reveal additional 
television channel buttons that are not currently 
displayed on the display screen. 

The on button 61, off button 62, and television 
channel button 66 - 69 objects available in the 
television Space 64 are not portals that give access 
to other Spaces in the Space map of Figure 1b. 
Instead, the on button 61, off button 62, and televi- 
sion channel buttons 66 - 69 are buttons that are 
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associated with functions. For example, the green 
"on" button 61 and red "off" button 62 are asso- 
ciated with the functions "turn the television on" 
and "turn the television off" respectively. When a 
user Selects either the on or off button, the button 
anticipates to provide feedback to the user that the 
button was Selected. The hand-held display device 
then transmits the appropriate commands that turn 
the television on or off. Similarly, if the user Selects 
any of the television channel buttons 66 - 69, the 
television channel button anticipates and the hand- 
held display device transmits the appropriate com- 
mands to the real television thereby tuning the 
television to the Selected channel. The process of 
transmitting commands to the physical television 
set is explained in detail in the following section. 

Referring back to Figure 7, if a user Selects 
the television program schedule book portal 71, the 
television program schedule book portal 71 antici- 
pates, and the user interface moves the user to the 
television program schedule Space. Figure 9 de- 
picts the television program schedule book Space 
of the present embodiment. In the television pro- 
gram schedule book Space of Figure 9, the user is 
presented with a two dimensional array of televi- 
sion program listings. The array organizes the tele- 
vision, program listings vertically by television chan- 
nel and horizontally by time. Each television pro- 
gram listing is a portal that can be Selected by the 
user. Each television program listing can also be 
carried by the user and Dropped onto the video 
cassette recorder portal such that the hand-held 
display device programs the video cassette re- 
corder to record the television program. 

The television program schedule book Space 
of Figure 9 is actually organized as a pair of 
connected vertically oriented object wheels. Refer- 
ring to Figure 11, a conceptual diagram of the 
television program schedule book Space is illus- 
trated. The television channels of Figure 9 are 
placed on a first object wheel 72 as illustrated in 
Figure 11. The television program listings are 
placed a second object wheel 85. The two object 
wheels 72 and 85 are connected such that if the 
user Spins either object wheel, the other object 
wheel will spin at the same rate such that the 
television channels and television program listings 
are always aligned. 

Each television program listing on the televi- 
sion program schedule object wheel 85 is a portal. 
When a user Selects one of the television program 
listing portals, the television program listing portal 
anticipates and the user interface moves the user 
to a television program information Space. Figure 
10 illustrates a typical television program informa- 
tion Space. The television program information 
Space includes a clock 81, a date 82 and television 
channel 83 which inform the user when the televi- 



sion program is on and what channel the television 
program is on. The television program information 
Space also includes a description of the television 
program as illustrated in Figure 10. 

5 

The Agent 

The Agent is an anthropomorphic character 
designed to help a user of the present invention in 
w a number of ways. The Agent assists users by 
providing help, providing state information, alerting 
the user to certain events, and acting as a reposi- 
tory. Each function performed by the Agent will be 
explained individually. 

/5 

HeJp 

As previously described in the tour of the 
present embodiment, with reference to Figure 3, 

20 when a user first enters the user interface of the 
present invention, the Agent demonstrates how to. 
use the Point gesture and the Select gesture. Fur- 
thermore, the Agent demonstrates how to use the 
Pan gesture to move around within a Space. The 

25 Agent can also demonstrate how to use other fea- 
tures of the user interface as necessary. 

To further aid users, the Agent carries a help 
manual in its briefcase that is available to the user 
at any time. To access the help manual, the user 

30 simply Selects the Agent portal. The Agent antici- 
pates by providing ah animation of the Agent open- 
ing up its briefcase. The user then Selects the help 
manual from within the Agent's briefcase. 

35 State and Mode Information 

The Agent is also used to express the current 
state or mode of the user interface. To express the 
current state or mode the Agent changes its cloth- 

40 ing, changes its expression, or changes the props 
it is carrying. 

For example, if the user interface of the 
present invention is placed into a diagnostic mode, 
the clothing that the Agent is wearing is changed to 

45 convey this fact. Referring to Figure 17a, the 
Agent is shown in a doctor's uniform to symbolize 
that the user interface is in a diagnostic mode. 

Figure 17b provides another example of how 
the Agent's appearance can convey state informa- 

50 tion. When the user is in a writing state, the Agent 
is given a pencil to symbolize that the user inter- 
face is in a writing state. 

Alerts 

55 

While a user is using the hand-held display 
device of the present invention, certain events may 
occur that require the user to be informed imme- 
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diately. Such events are usually referred to as 
alerts. For example, if the hand-held display device 
is coupled to an electronic messaging service, the 
hand-held display device should alert the user 
when a message has been received that is ad- 5 
dressed to the user. 

In the user interface of the present invention, 
the Agent is used to obtain the user's attention in 
order to inform the user about any alerts. To get 
the user's attention the Agent waves its hand. Addi- w 
tionally, if the. alert is very important, the Agent 
may wave both arms or produce sounds to further 
attract the user's attention. 

For example, when an event occurs that re- 
quires the user's attention, the Agent begins wav- "/s 
ing as illustrated in Figure 17c. To explain what 
caused the alert, the Agent's appearance may be 
further altered as demonstrated in the previous 
section in order to convey information about the 
specific alert. For example, when an urgent mes- 20 
sage arrives, the Agent can wave with one hand 
while holding a telegram in the other hand. 

Since alerts are generally more important than 
state or mode information, alerts will take prece- 
dence over state or mode information. Therefore, if 25 
the image of the Agent is currently being used to 
convey .state or mode information when an impor- 
tant event occurs, the image of the Agent will 
change to get the user's attention and inform the 
user about the alert. For example, if the Agent is 30 
holding a pencil while in writing mode, as illustrated 
in Figure 17b, when an urgent message arrives, 
the Agent will immediately change to the waving 
animation as illustrated in Figure 17c to inform the 
user about the message. 35 

Remote Device Control 

The hand-held display device of the present 
invention is designed to control remote devices. 40 
Accordingly, the user interface defined in the pre- 
vious section for the hand-held display device is 
designed to be an ideal graphical user interface for 
controlling remote devices. For example, to control 
a device in the user's living room, a user first 45 
moves to the living room Space 60 of the user 
interface as illustrated in Figure 7. The user then 
Selects an object corresponding to a device that 
the user wishes to control. For example, if the u?er 
desires to control a television, the user Selects 'ne 50 
television portal 73 associated with a real televi- 
sion. When the user Selects the television portal 
73, the television portal 73 anticipates and then 
displays a graphical user interface for controlling 
4he physical television in the user's living room. 55 
Figure 8 illustrates one possible embodiment of a 
set of television controls for controlling a television. 
The user then controls the television using the set 



of television controls displayed on the hand-held 
display device. 

In order for a remote device to be controlled by 
the hand-held display device, the hand-held display 
device must have certain information about the 
remote device. Specifically, the hand-held display 
device must be provided with an object-oriented 
programming type of object that is associated with 
the remote device. The object comprises a plurality 
of methods including a method for displaying the 
object on the display screen of the hand-held dis- 
play device and a method for displaying a graphi- 
cal user interface for the remote device. 

To simplify the operation of the present inven- 
tion, each compatible remote device automatically 
transmits the required object-oriented programming 
type of object to the hand-held display device. 
When a remote device transmits the object to the 
hand-held display device, the hand-held display 
device invokes the method for displaying the object 
on the display screen. The user can then move the 
object to an appropriate Space. If the user Selects 
the object the Selected object anticipates and then 
the method for displaying the graphical user inter- 
face is invoked. The method displays the remote 
device's graphical user interface on the hand-held 
display device's touch screen display. The user 
can then control the remote device by interacting 
with the remote device's graphical user interface 
displayed on the touch screen display. 

The hand-held display device of the present 
invention automatically obtains a graphical user in- 
terface from two types of compatible remote de- 
vices. The first type of compatible remote device is 
the intelligent remote device with a dedicated com- 
puter control system. The second type of compati- 
ble remote device is a simple remote device that 
can only respond to simple coded commands. 

The hand-held display device interacts differ- 
ently with the two types of compatible remote 
devices. The hand-held display device commu- 
nicates with intelligent remote devices using peer- 
to-peer communication with remote method invoca- 
tion for remote device control. The hand-held dis- 
play device communicates with simple remote de- 
vices using only simple control codes. However, 
from the user's perspective the differences be- 
tween the intelligent remote devices and the simple 
remote devices cannot be seen, and the user's 
interaction with the graphical user interface is iden- 
tical for both types of remote devices. The im- 
plementation detail of whether a user's actions 
cause a remote method invocation, or transmission 
of control codes through some communication 
mechanism is hidden from the user. 

This section will explain how the two types of 
compatible remote devices transmit information to 
the hand-held display device. Furthermore, a third 
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type of remote device will explained. The third type 
of device consists of any electrical device that can 
be controlled by coded infrared signals. 

Within this section, reference will be made to 
object-oriented programming type objects. Each 
object-oriented programming type of object con- 
tains data and methods that can be invoked to 
perform functions. To simplify the terminology 
used, the term "program object" will be used when 
referring to an object-oriented programming type of 
object. To eliminate confusion, the objects dis- 
played on the touch screen display in the user 
interface disclosed in the previous user interface 
will be referred to as "display objects" in this 
section. 

Peer to Peer Communication With Remote 
Method Invocation 

In the present invention, a remote device is 
defined as an "intelligent remote device" if that 
remote device contains a microprocessor, a com- 
munication system, and a sufficient amount of 
memory such that it can invoke methods in a 
program object. Figure 1c illustrates, in block dia- 
gram form, the components in an intelligent remote 
device. The hand-held display device of the 
present invention communicates with intelligent re- 
mote devices in a peer-to-peer fashion. 

The intelligent remote device has an asso- 
ciated graphical user interface comprising touch 
screen control elements such as dials, switches, 
sliders, etc. The intelligent remote device's graphi- 
cal user interface is defined within a user interface 
program object 303 stored in the intelligent remote 
device's memory 404. When an intelligent remote 
device is first turned on, the intelligent remote 
device automatically transmits its user interface 
program object 303 to the hand-held display de- 
vice. The hand-held display device can then dis- 
play the user interface for the intelligent remote 
device as defined by the user interface program 
object. 

When a user interacts with the intelligent re- 
mote device's graphical user interface as displayed 
on the hand-held display device, the hand-held 
display device invokes methods in a device driver 
program object 301 within the intelligent remote to 
control the remote device. Thus the software that 
actually "controls" an intelligent remote device is 
located within the intelligent remote device itself. 

The procedure of transmitting the user inter- 
face object 303 from an intelligent remote device to 
the hand-held display device will be disclosed with 
reference to Figures 18a - 18d, 19a - 19d and the 
flowchart in Figure 20. Figures 18a - 18d illus- 
trate, on a high level, how the intelligent remote 
device 150 and the hand-held display device 170 



of the present invention communicate with each 
other. Figures 19a - 19d illustrate, in conceptual 
form, how the user interface program object 303 in 
the intelligent remote device 150 is transmitted to 
5 the hand-held display device 170. 

Referring to step 101 of Figure 20, when a 
new intelligent remote device 150, such as the 
intelligent thermostat control device in Figure 18a, 
is first turned on, the new intelligent remote device 

w 150 broadcasts across the communications net- 
work 160 that it has a user interface program 
object to export. As set forth in the hardware sec- 
tion, the communication on the communications 
network 160 can take place using radio waves, low 

/s power, cellular, infrared signals, or electrical signals 
on existing power lines in the case of an Echelon® 
base LON® system. In step 102, a hand-held dis- 
play device 170 receives the interface export mes- 
sage. The hand-held display device 170 signals 

20 back across the communications network 160 to 
the new intelligent remote device 150 requesting 
the new intelligent device 150 to begin transmitting 
the user interface object 303. 

The intelligent remote device 150 then trans- 

25 mits the user interface object 303 to the display 
device 170 as specified in step 103 of Figure 20 
and illustrated in Figure 18a. Referring to Figure 
19a, the intelligent remote device 150 also trans- 
mits a pointer (as indicated by dotted lines) to a 

30 device driver program object 301 within the intel- 
ligent remote device 150. The user interface pro- 
gram object 303 defines how the user interface for 
the intelligent remote device 150 should appear on 
a display screen. The user interface program ob- 

35 ject 303 invokes methods in a device driver pro- 
gram object 301 to handle user interface inter- 
actions. The device driver program object 301 con- 
tains methods that control the actual intelligent 
remote device 150. Referring to Figure 19b, the 

40 display device 170 now has a copy of the intel- 
ligent device's user interface program object 313 
and a device driver program object 311 pointer. 

After receiving the user interface program ob- 
ject 313, the hand-held display device 170 invokes 

45 a method in the user interface program object 313 
that renders a display object in the current Space 
as stated in step 104. The new intelligent device 
150 also registers with a remote name server con- 
nected to the communications network 160. 

so When a user Selects the display object asso- 

ciated with the new intelligent remote device 150, 
the hand-held display device 170 invokes a method 
in the user interface program object 313 as stated 
in step 105 (the object with an invoked method is 

55 shown in bold). The method invoked in the user 
interface program object 313 displays the remote 
intelligent device's graphical user interface on the 
display screen 171 of the display device 170 as 

19 
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illustrated in Figure 18b. A user can interact with 
the remote intelligent device's graphical user inter- 
face displayed on the display device 170 as stated 
in step 106. 

The user controls the intelligent remote device 
150 by interacting with the graphical user interface 
displayed on the hand-held display device 170. To 
resolve the user's interactions, the user interface 
program object 313 requires control methods in the 
intelligent remote device's device driver program 
object 301. To access the control methods in the 
intelligent remote device's device driver program 
object 301, the hand-held display device 170 uses 
the device driver program object pointer 311. The 
device driver program object pointer 311 invokes 
the methods in the real device driver program 
object 301. The intelligent remote device's device 
driver program object 301 is invoked by sending a 
message over the communication network 160 as 
illustrated in Figures 18c and 19c (again, the 
object with an invoked method is shown in bold). 
The methods invoked in the intelligent remote de- 
vice's device driver program object 301 control the 
intelligent device 150 as stated in step 108. 

Finally, in step 109, any return values or feed- 
back generated by the device driver program ob- 
ject 301 in intelligent remote device 150 are sent 
back to the hand-held display device 170 over the 
communications network 160 as illustrated in Fig- 
ures 18d and 19d. The user interface program 
object 313 changes the graphical user interface on 
the display screen 171 in response to the return 
values or feedback received from device driver 
program object 301 in the intelligent remote device 
150. 

A robust communication protocol across the 
communications network 160 is required to com- 
municate remote method invocations between the 
hand-held display device 170 and the intelligent 
remote device 150. Both the hand-held display 
device 170 requesting the method invocation and 
the intelligent remote device 150 that executes the 
method must be able to handle a broken commu- 
nication path. Therefore, both the hand-held display 
device 170 and the intelligent remote device 150 
have a dedicated remote procedure call (RPC) 
manager that handles the remote method invoca- 
tion. Referring to Figures 19a - 19d, the remote 
procedure call manager 309 transmits the remote 
method invocation request for the hand-held dis- 
play device 170. On the other side, a server remote 
procedure call manager 307 processes the re- 
ceived remote method invocation in an intelligent 
remote device 150. 

Figures 21a and 21b illustrate state diagrams 
used to control the remote procedure call (RPC) 
managers that handle the remote method invoca- 
tion on both the hand-held display device 170 side 



and the remote device side. Taken together, state 
diagrams in Figures 21a and 21b disclose a hand- 
shake used between the hand-held display device 
170 and an intelligent remote device 150 to ensure 
5 reliable communications. 

Figure 21a illustrates a state machine for the 
remote procedure call manager 309 in the hand- 
held display device 170 that handles the display 
device's side of a remote method invocation. In the 

w present invention, the user interface program object 
313 in the hand-held display device 170 requests 
remote method invocations to the device driver 
program objects 301 located in remote intelligent 
devices. Referring to Figure 21a, when a user 

75 interface object requests a remote method invoca- 
tion, the hand-held display device's RPC manager 
enters the "ready to transmit" state 403. The dis- 
play device's RPC manager then sets ah error 
countdown timer and transmits a packet containing 

20 a message with a method selector and associated 
data. After transmitting the packet, the display de- 
vice's RPC manager enters a "waiting for reply" 
state 405. If the proper reply message is sent back, 
the method invocation is complete and the reply 

25 message is analyzed at state 407. 

Remote method invocations do not always 
complete so easily. If the device driver program 
object 301 in the intelligent remote device 150 
needs more time to complete the method, it will 

30 send back a message requesting more time. The 
display device's RPC manager responds by mov- 
ing to state 409 where it increases the timeout 
value. If the display device's RPC manager does 
not receive a reply by the time the error countdown 

35 timer expires, the display device's RPC manager 
moves to an error recovery state 411. If the error 
countdown timer expires less than a predetermined 
number of times, the display device's RPC man- 
ager returns to the "ready to transmit" state 403. If 

40 the error countdown timer reaches a predetermined 
limit, the display device's RPC manager abandons 
the remote method invocation and enters a failed 
state 413. The user is then usually informed of the 
communications problem. 

45 Figure 21b illustrates a state machine for a 

remote procedure call manager 307 in an intelligent 
remote device that handles the remote device's 
side of a remote method invocation. The remote 
device's RPC manager 307 starts in a "waiting to 

so receive" state 451. When a packet is received, the 
remote device's RPC manager analyzes the packet 
at state 453. If the received packet contains a new 
message requesting a method invocation, the re- 
mote device's RPC manager gives the packet to an 

55 object that performs the method of the request at 
state 455. The remote device's RPC manager then 
transmits a result and returns to the waiting state 
451. 
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If the received packet contains a message that 
the remote device has already seen, the RPC man- 
ager of the intelligent remote device moves to state 
457 where the intelligent remote device determines 
whether the method invocation request was already 
completed or whether it is still working on the 
request. If the intelligent remote device has already 
completed the request, the subsystem retransmits 
the results at state 459 and returns to the waiting 
state 451. Otherwise, if the intelligent remote de- 
vice is still working on the method invocation, the 
remote device's RPC manager moves to state 461 
where the remote device's RPC manager requests 
more time to complete the invoked method and 
then returns to the waiting state 451. 

Figure 21c illustrates the sequence of a re- 
mote method invocation where there are no com- 
munication problems. At step 501, a user interface 
object in the hand-held display device requests a 
remote method invocation. Therefore, the user in- 
terface object sends a message with a method 
selector and argument to the display device's RPC 
manager. The display device's RPC manager 
packs, the message received into a packet at step 
502 and transmits the packet across the commu- 
nications network at step 503. 

The intelligent remote device's RPC manager 
receives the transmitted packet at step 504. The 
remote device's RPC manager unpacks the mes- 
sage from the packet at step 505 and directs the 
packet to the device driver program object within 
the intelligent remote device. The device driver 
program object invokes the method specified by 
the method selector in the message at step 506. 
The device driver program object generates a re- 
turn message that is passed back to the remote 
device's RPC manager. At step 507, the remote 
device's RPC manager packs the return message 
into a reply buffer. The remote device's RPC man- 
ager sends the reply buffer at step 508. 

The display device's RPC manager receives 
the reply buffer at step 509 and unpacks the reply 
buffer to extract a return value at step 510. The 
display device's RPC manager delivers the return 
value to the requesting user interface program ob- 
ject that invoked the method in the device driver 
program object. Finally, the user interface program 
object analyzes the return value at step 511. 

As illustrated, the present invention can control 
an intelligent remote device by remotely invoking 
methods in a device driver program object located 
within the intelligent remote device. In addition, it 
will be appreciated by the reader that the remote 
device may also remotely invoke methods in the 
display device. For example, the device driver ob- 
ject in a remote device may invoke a method in a 
user interface object in the display device to dis- 
play a message. 



Many remote devices are not sophisticated 
enough to have interna! control methods that can 
be invoked by the display device. Therefore, the 
present invention provides other methods for con- 
s trolling simple devices. 

Simple Device Control 

In the present invention, a remote device is 
io defined as a "simple device" if the remote device 
does not have a microprocessor or the remote 
device contains such a primitive processor that the 
display device can not invoke control methods 
within the remote device. Figure 1d illustrates, in 
is block diagram form, the components in a simple 
remote device. To control a simple device, the 
hand-held display device of the present invention 
transmits simple predefined control codes instead 
of invoking control methods within the simple re- 
20 mote device. Therefore, to control a simple remote 
device, the hand-held display device must obtain 
both a user interface program object and a device 
driver program object from the simple remote de- 
vice. 

25 An example of obtaining user interface pro- 

gram object and a device driver program object 
from a simple remote device is given with refer- 
ence to Figures 22a - 22d, 23a - 23d and the 
flowchart in Figure 24. Figures 22a - 22cl illustrate 

30 on a high level how the remote simple device 155 
and the hand-held display device 170 of the 
present invention communicate with each other. 
Figures 23a - 23d illustrate, in conceptual form, 
how the user interface program object 353 and the 

35 device driver program object 351 of the remote 
external device 150 are transported the hand-held 
display device 170 and used. 

Referring to Figure 22a, a thermostat imple- 
mented as a simple remote device 155 is illus- 

40 trated. As specified by step 124 of Figure 24, the 
simple remote device 155 o,f Figure 22a transmits 
both a user interface program object and a device 
driver program object to the hand-held display de- 
vice 170. Illustrated conceptually in Figure 23a, the 

45 simple device 155 transmits the user interface pro- 
gram object 353 and the device driver program 
object 351. The transmission of the program ob- 
jects is done using some communications means 
162 that may be a communications network or any 

so other means of transferring the information into the 
display device 170. 

The hand-held display device 170 receives the 
copy of the user interface program object 363 that 
describes the simple device's graphical user inter- 

55 face. The hand-held display device 170 then in- 
vokes a method in the user interface program 
object 363 that renders a display object within a 
Space shown on the hand-held display device 170. 
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When a user Selects the display object associated 
with the simple remote device 155, the display 
device 170 invokes a method within the user inter- 
face program object 363 as stated in step 125 of 
Figure 24 and illustrated in Figure 22b. The meth- 
od invoked in the user interface program object 
363 displays a graphical user interface associated 
with the simple remote device on the display 
screen 171 of the display device 170. The user can 
then interact with simple remote device's graphical 
user interface on display device 170 as stated in 
step 127. 

The user's interactions with the simple device's 
graphical user interface are analyzed by the simple 
device's device driver program object 361 within 
the display device 150. The device driver program 
object 361 resolves the user's interactions into con- 
trol codes that the simple remote device 155 will 
respond to. As illustrated in Figure 23c, the device 
driver program object 361 located within the dis- 
play device 170 analyzes the user's actions. 

Referring to Figures 22c and 23c, the hand- 
held display device 170 transmits the control codes 
to the simple remote device 155. The simple re- 
mote device 155 changes in response to the con- 
trol codes received as stated in step 130. Finally, if 
the simple remote device 155 and the display 
device 170 are coupled together by a communica- 
tions network, the simple device 155 can transmit 
any return values or feedback generated by the 
simple remote device 155 as illustrated in Figure 
22d. 

Conventional Device Control 

Existing remote devices that are not designed 
for the hand-held display device of the present 
invention do not have a user interface program 
object to transmit to the hand-held display device. 
For example, Figure 1e illustrates a typical existing 
remote device that can only respond to coded 
command signals such as infrared signals. In such 
cases, the hand-held display device must be man- 
ually informed about the remote device. The hand- 
held display device then uses a generic user inter- 
face program object for the type of remote device. 

For example, if a user has a conventional tele- 
vision that is controlled by infrared signals, the user 
enters a model number of the television to inform 
the hand-held display device about the conven- 
tional television. The hand-held display device then 
accesses a database to obtain a list of the control 
codes that the conventional television will under- 
stands. A television display object is then created 
to control the conventional television. If the user 
selects television display object, the hand-held dis- 
play device will invoke a method that displays a 
generic television user interface. When the user 



interacts with the generic television user interface, 
the display device will transmit the proper infrared 
control codes to control the television. 

s Summary 

The foregoing has described an intuitive 
graphical user interface based upon a geographic 
map structure and a system for controlling remote 

/o external electronic devices. In the user interface of 
the present invention, each Space of the geo- 
graphic map structure is rendered on a touch 
screen display as a Space. Within each Space are 
colored icons called "objects" that can be Selected 

/5 and manipulated by the user. Certain objects, re- 
ferred to as portals, transport the user from one 
Space to another Space when Selected. Other ob- 
jects, referred to as buttons, perform associated 
actions or functions when Selected. The user inter- 

20 face is displayed on a hand-held display device 
that can control remote electronic devices. Each 
remote electronic device transmits a user interface 
program object that defines a graphical user inter- 
face to the display device. The hand-held display 

25 device displays the graphical user interface asso- 
ciated with the remote device when a user selects 
an icon associated with the remote device on the 
display device's touch screen display. It is con- 
templated that changes and modifications may be 

30 made by one of ordinary skill in the art, to the 
materials and arrangements of elements of the 
present invention without departing from the scope 
of the invention. 

35 Claims 

1. A method of providing a graphical user inter- 
face on a computer system, said computer 
system comprising a CPU, a memory, and 
40 graphics display screen, said method compris- 

ing the steps of: 

providing a plurality of Spaces, each of 
said Spaces comprising a graphic background 
image; 

45 organizing said Spaces in a directed graph 

structure, said Spaces connected together with 
a plurality of portals; 

locating said plurality of portals in said 
Spaces, each of said portals having an asso- 
50 ciated target Space such that said portal con- 

nects the Space said portal is in with said 
associated target Space, each of said portals 
comprising a graphic icon; 

rendering a graphic background image of 
55 a first Space and portals located in said first 

Space on said graphics display screen, said 
first Space having at least a first portal; 

receiving input from a user, said input 
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comprising a portal selected from said portals 
located in said first Space; and 

rendering a graphic background image of 
a target Space associated with said selected 
portal on said graphics display screen; 5 
such that said user can move between Spaces 
in said directed graph structure by selecting 
said portals within said Spaces. 

The method of providing a graphical user inter- w 
face on a computer system as claimed in 
claim 1 further wherein said steps of rendering 
a background image further comprises the 
steps of: 

rendering a first subset of said background rs 
image on said graphics display screen, said 
first subset from a first location in said back- 
ground image; 

receiving directional input from said user, 
said directional input comprising a movement 20 
direction; and 

rendering a second subset of said back- 
ground image on said graphics display screen, 
said second subset from a second location in 
said background image, said second location 25 
being said first location plus an offset cor- 
responding to said movement direction; 
such that said user can move about within a 
background image associated with a Space. 

30 

The method of providing a graphical user inter- 
face on a computer system as claimed . in 
claim 2 further wherein said step of receiving 
directional input from said user further com- 
prises detecting a finger of said user sliding 35 
across said display screen. 

The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1 further comprising the steps of: 40 

providing a Wayback portal, said Wayback 
portal displayed in each of said Spaces, said 
Wayback portal comprising a stack of data 
structures, each of said data structures com- 
prising a pointer, each of said pointers pointing 45 
to one of said Spaces; 

pushing a first data structure on said 
Wayback portal stack when said user moves to 
said first Space, said first data structure con- 
taining a pointer to said first Space; 50 

popping a second data structure off of said 
Wayback portal stack when said user selects 
said Wayback portal, said second data struc- 
ture containing a pointer to a second Space; 
and 55 

moving said user to said second Space 
pointed to by said pointer in said second data 
structure popped off said Wayback portal 



stack; ■ 

such that said user can trace back along a 
path of Spaces which said user travels 
through. 

5. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 4 wherein said data structures in said 
Wayback portal further comprises a coordi- 
nate, said coordinate identifying a location in 
said first Space where said user was when 
said user left said first Space. 

6. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 4 wherein said Wayback portal is ren- 
dered as a snapshot of a Space pointed to by 
a pointer in a data structure on top of said 
Wayback portal stack. 

7. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1 further comprising the steps of: 

providing a plurality of button objects, 
each of said button objects associated with a 
function; 

rendering said button objects within said 
Spaces, said button objects comprising graph- 
ic icons; and 

executing the function associated with a 
button object when said button object is se- 
lected by a user. 

8. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1 wherein said display screen comprises 
a touch sensitive display screen. 

9. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 8 wherein said input from said user 
comprises a finger touching said touch sen- 
sitive display screen. 

10. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1, said method further comprising the 
steps of: 

rendering an anthropomorphic graphic 
character on said display; 

attracting said user's attention by animat- 
ing said anthropomorphic graphic character; 
and 

conveying state information to said user by 
changing a visual appearance of said anth- 
ropomorphic character. 
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11. A method of displaying a plurality of objects 
for a user to select from on a display screen 
associated with a computer system, said meth- 
od comprising the steps of: 

organizing a plurality of objects into a cir- 
cular object list; 

rendering a first subset of said objects in 
said circular object list as a first subset of 
graphic objects on said display screen, said 
graphic objects rendered on said display 
screen aligned along a first axis; and 

rotating through said circular object list 
when said user slides a pointing device across 
said graphic objects in a direction substantially 
parallel to said first axis such that said first 
subset of graphic objects on said display 
screen is replaced by a second subset graphic 
objects on said display screen, said second 
subset graphic objects on said display screen 
corresponding to a second subset of said ob- 
jects in said circular object list; 
such that said user can choose any of said 
plurality of objects by rotating through the cir- 
cular object list until a desired object appears 
on said display screen. 

12. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 1 1 further comprising the step of: 

selecting a graphic object displayed on 
said display screen when a user places a 
pointing device on said graphic object dis.- 
played on said display screen for a minimum 
amount of time. 

13. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 11 further comprising the step of: 

moving a graphic object off said circular 
object list when said user places said pointing 
device on said graphic object and moves said 
pointing device in a direction substantially per- 
pendicular to said first axis. 

14. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 13 wherein said display screen com- 
prises a touch screen and said pointing device 
comprises a finger. 

15. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 1 1 wherein said step of rotating through 
said circular object list is enhanced by simulat- 
ing a physical wheel that is affected by friction. 

16. In a graphical user interface for a computer 
system, said graphical user interface compris- 



ing a plurality of Spaces organized in a di- 
rected graph structure, each of said Spaces 
comprising a graphic background image, said 
Spaces coupled in said directed graph struc- 

5 ture using a plurality of portals, each of said 

portals proviiding a connection between a first 
Space and a second Space, a method for 
storing a path traveled along said directed 
graph structure, said method comprising the 

10 steps of: 

rendering a graphic background image 
corresponding to a first Space on a display 
screen, said first Space including at least one 
portal, said portal associated with a target 

75 Space; 

providing a Wayback portal, said Wayback 
portal displayed in each of said Spaces, said 
Wayback portal comprising a stack of Space 
pointers, each of said Space pointers pointing 

20 to a previously visited Space; 

receiving input from a user, said input 
comprising said Wayback portal or a portal 
selected from said portals located in said first 
Space; and 

25 processing said input received from said 

user such that 

if said user selects a portal, said user 
interface pushing a pointer to a Space asso- 
ciated with said selected portal on said 

30 Wayback stack and rendering a graphic back- 

ground image of the Space associated with 
said selected portal on said graphics display 
screen, else 

if said user selects said Wayback portal, 

35 said user interface popping a Space pointer off 

the top of said Wayback portal stack and ren- 
dering said Space indicated by said Space 
pointer on said graphics display screen. 

40 17. The method of storing a path traveled along 
said directed graph structure in a graphical 
user interface for a computer system as 
claimed in claim 11 wherein said Wayback 
portal comprises a miniature image of the 

45 Space indicated by the pointer on the top of ■ 

the Wayback portal stack. 

18. In a computer system having a graphical user 
interface, an animated multipurpose icon, said 
50 icon comprising the elements of: 

an anthropomorphic graphic character ren- 
dered on said display; 

means for attracting a user's attention by 
animating said anthropomorphic graphic char- 
55 acter; and 

means for conveying state information by 
changing a visual appearance of said anth- 
ropomorphic character. 
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19. In a graphical user interface for a computer 
system, said graphical user interface compris- 
ing a plurality of graphic objects rendered on a 
display screen, said display screen comprising 
a touch screen which senses a finger placed 
on the screen and determines a location of 
said finger on said touch screen, a method of 
allowing a user to grab a graphic object, said 
method comprising the steps of: 

rendering a graphic object on said touch 
screen display, said graphic object defining an 
area on said touch screen display; 

placing a finger on said touch screen dis- 
play near said graphic object rendered on said 
touch screen display; 

circling said graphic object rendered on 
said touch screen display with said finger; 

touching said graphic object rendered on 
said touch screen display; and 

rendering said graphic object on said 
touch screen display at a location defined by 
said finger on said display screen as deter- 
mined by said touch screen display; 
such that a user can select said graphic object 
by circling said graphic object with a finger. 

20. An apparatus for controlling at least one re- 
mote device, said apparatus comprising the 
elements of: 

a central processing unit (CPU); 

a memory/ said memory coupled to said 
central processing unit; 

a display screen, said display screen 
coupled to said CPU, said display screen dis- 
playing graphic images; 

communications means for receiving and 
transmitting messages across a communica- 
tions network, said communications means 
coupled to said central processing unit; 

means for receiving a user interface pro- 
gram object over said communications means, 
said user interface program object broadcast 
by a remote device, said user interface pro- 
gram object defining a user interface for said 
remote device; 

means for invoking a user interface meth- 
od in said user interface program object, said 
user interface method displaying a graphical 
user interface associated with said remote de- 
vice on said display screen; 

means for accepting input from a user, 
said input guided by said graphical user inter- 
face associated with said remote device on 
said display screen; and 

means for invoking a device driver method 
associated with said remote device in re- 
sponse to said input from said user, said de- 
vice driver method controlling said remote de- 



vice; 

such that said user interface of said remote 
device is displayed on said display screen and 
interaction with said user interface invokes a 
5 device driver method. 

21. The apparatus for controlling at least one re- 
mote device as claimed in claim 20, said ap- 
paratus further comprising 

w means for receiving a device driver pro- 

gram object associated with said remote de- 
vice over said communications means, said 
device driver program object including said 
device driver method for controlling said re- 

15 mote device. 

22. The apparatus for controlling at least one re- 
mote device as claimed in claim 21, said ap- 
paratus further wherein said device driver 

20 method is broadcast by said remote device 

across said communications network and 
copied into said apparatus. 

23. The apparatus for controlling at least one re- 
25 mote device as claimed in claim 22 wherein 

said device driver method translates said input 
from said user into a plurality of control codes. 

24. The apparatus for controlling at least one re- 
30 mote device as claimed in claim 23 wherein 

said apparatus transmits said plurality of con- 
trol codes to said remote device. 

25. The apparatus for controlling at least one re- 
35 mote device as claimed in claim 20 wherein 

said device driver method is located* within 
said remote device. 

26. The apparatus for controlling at least one re- 
40 mote device as claimed in claim 25 wherein 

said device driver method is invoked remotely 
across said communications network by said 
apparatus. 

45 27. The apparatus for controlling at least one re- 
mote device as claimed in claim 26 wherein 
said device driver method is remotely invoked 
using a robust remote procedure call manager. 

so 28. A method of obtaining a graphical user inter- 
face from said remote device and controlling 
said remote device with a local display device, 
said local display device coupled to a commu- 
nication network, said method comprising: 

55 connecting said remote device to said 

communication network; 

signaling from said remote device that said 
remote device has a graphical user interface to 



25 



49 



EP 0 626 635 A2 



50 



export upon being connected to said commu- 
nication network; 

requesting said graphical user interface by 
said local display device; 

transmitting a graphical user interface pro- 
gram object to said local display device, said 
graphical user interface program object con- 
taining a method which display said graphical 
user interface; and 

invoking said method in said user interface 
program object such that said remote device's 
graphical user interface is displayed on said 
local display device. 

29. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 28, wherein 
said method further comprises the steps of; 

interacting with said remote device's user 
interface displayed on said local display de- 
vice; and 

invoking a control method in a device driv- 
er program object to handle said interactions 
with said remote device's user interface. 

30. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 29, wherein 
said device driver program object is located 
within said remote device. 

31. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 30, wherein 
said control method in said device driver pro- 
gram object is invoked remotely across said 
communications network. 

32. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 29, said meth- 
od further comprising the step of transmitting 
said device driver program object across said 
communications network to said local display 
device. 

33. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 32, wherein 
said device driver method within said device 
driver program object is invoked within said 
local display device. 



34. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 33, wherein 

5 said device driver method translates said user 

input into commands codes. 

35. The method of obtaining a remote device's 
user interface from said remote device and 

w controlling said remote device with a local dis- 

play device as claimed in claim 34, wherein 
said local display device transmits said com- 
mands codes to said remote device. 

/5 36. A method of copying an object in a graphical 
user interface, said method comprising the 
steps of: 

rendering a first graphic image of said 
object at a first position on a graphics display 
20 screen; 

receiving input from a user, said input 
comprising a command to copy said object; 

rendering a ghost image of said object at 
said first position on said display screen, said 
25 ghost image comprising a copy of said first 

graphic image of said object drawn in muted 
colors; 

receiving input from a user, said input 
comprising a second position to locate a copy 
30 of said object; 

rendering said first graphic image of said 
object at said second position; and 

changing the ghost image of said object at 
said first position on said display screen such 
35 that said ghost image of said object becomes 

identical to said first graphic image of said 
object; 

such that two graphic images of said object 
now exist. 

40 

37. The method of copying an object in a graphi- 
cal user interface as claimed in claim 36, 
wherein said input comprises a finger touching 
a touch screen display 

45 

38. A method of moving an object in a graphical 
user interface, said method comprising the 
steps of: 

rendering a first graphic image of said 
so object at a first position on a graphics display 

screen; 

receiving input from a user, said input 
comprising a command to move said object; 

rendering a ghost image of said object at 
55 said first position on said display screen, said 

ghost image comprising a copy of said first 
graphic image of said object drawn in muted 
colors; 
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receiving input from a user, said input 
comprising a second position to locate a said 
object; 

rendering said second graphic image of 
said object at said second position; and 5 

moving said ghost image of said object at 
said first position to said second position such 
that said ghost image of said object merges 
into said second graphic image of said object 
at said second position; io 
such that one graphic image of said object 
now exists at said second position. 

39. The method of copying an object in a graphi- 
cal user interface as claimed in claim 38, 15 
wherein said input comprises a finger touching 

a touch screen display 

40. In an object-oriented graphical user interface, 

said object-oriented graphic user interface il- 20 
lustrating objects as graphical icons, a method 
of providing information to a program object, 
said method comprising the steps of: 

rendering a graphic image of a data object 
at a first position on a graphics display screen; 25 

receiving input from a user, said input 
comprising a command to move said data ob- 
ject; 

rendering a ghost image of said object at 
said first position on said display screen, said 30 
ghost image comprising a copy of said first 
graphic image of said object drawn in muted 
colors; 

receiving input from a user, said input 
comprising a second position to locate a said 35 
data object, said program object located at 
said second position; and 

receiving input from a user, said input a 
command to provide said data object to said 
program object; 40 
such that two graphic images of said object 
now exist. 



user currently is within said map structure; 

displaying a plurality of portals in said first 
Space, each of said portals having an asso- 
ciated target Space such that said portal con- 
nects the Space said portal is in with said 
associated target Space such that each porta) 
provides said user a place to go, each of said 
portals comprising a graphic icon; and 

displaying a Wayback portal in said first 
Space, said Wayback depicting a image of a 
Space said user just left. 

43. The method of navigating through a graphical 
user interface as claimed in claim 42, wherein 
said map structure is organized as a directed 
graph. 

44. The method of navigating through a graphical 
user interface as claimed in claim 43, wherein 
if a user selects one of said portals then said 
user is moved to a Space associated with said. 
Portal. 



41. The method of providing information to a pro- 
gram object as claimed in claim 40, wherein 
said input comprises a finger touching a touch 
screen display 



45 



42. A method of navigating through a graphical 

user interface, said graphical user interface 50 
displayed on a display screen, said graphical 
user interface comprising a plurality of Spaces, 
each of said Spaces comprising a graphic 
background image, said Spaces organized into 
a map structure, said method comprising the 55 
steps of: 

displaying a first Space on said display 
screen, said first Space representing where a 



27 



BP 0 626 635 A2 



170 




INPUT/OUTPUT 



I 



■34 



EXTERNAL 
COMMUNICATION 



"35 



f 

VCR 



| I PLAY 



I □ 1||<3<|K o lll>o|irin 



•42 



OTHER INPUT 
DEVICES 

(MOUSE, 
TRACKBALL, 
JOYSTICK) 



-37 



TOUCH 

SCREEN 

DISPLAY 




150 



Figure lb 



29 



EP 0 626 635 A2 



401 



403 



404 



Communication 
System 



CPU 



A 



Memory 



303 



-150 



Device Control 



405 



Figure lc 



420 424 







1 1 

Emitter 





' Memory 




-< — 




1 




353 y^5^ 


















— >> 


Receiver 

1 i 1 


> Device Control 

i Circuitry 



-155 



421 425 

Figure Id 



450 455 





















> 




Device Control 




— >■ 






Circuitry 









-157 



Figure le 

(Prior Art) 



30 



EP 0 626 635 A2 



EARTH 




ANOTHER TOWN 





31 




32 




33 



EP 0 626 635 A2 




EP 0 626 635 A2 




35 



EP 0 626 635 A2 




36 



EP 0 626 635 A2 




Pink Panther Parade 







U.S. Government 




o 




Murder,She Spoke 

While Jessico (Angela Lansbury)is recording one 
of her novels for the Mystery Books for the Blind 
series, a murder occurs With Michael Cole, 
Charlie Dam els and Jonna Lee . 

%^ 

/ 45 

) 





37 



EP 0 626 635 A2 



PROGRAMS 



\ 

85 



Figure 11 



38 



EP 0 626 635 A2 




39 




EP 0 626 635 A2 




40 




41 



EP 0 626 635 A2 




42 



EP 0 626 635 A2 




43 



EP 0 626 635 A2 



Figure 17a 



Figure 17b 



Figure 17c 



44 



EP 0 626 635 A2 



150- 



THERMOSTAT 



404 
I 



— (MEMORY) 



1 



403 






\ 


CPU 



401 



COMM 




Figure 18a 



Figure 18b 



150 




Figure 18c 



150 



THERMOSTAT 



I 



403 



401 



(memory) 



CPU 



COMM 




Figure lSd 



45 



EP 0 626 635 A2 



150— 
301- 



A 



,303 



160 



t 



307 309 

Figure 19 a 



— 170 



150 — 
301- 



A 



150 — 



301- 



303 



313 



\ 

160 



a 



t 



307 309 

Figure 19b 



^303 



313 



160 



t 



307 



309 



Figure 19c 



150— 



301- 



,,303 



313 



\ 

160 



t 



307 



309 



Figure 19d 



— 170 
-311 



IV .2 



— 170 
-311 



— 170 
-311 



46 




EP 0 626 635 A2 



REMOTE DEVICE IS TURNED ON 
AND BEGINS ANNOUNCING 
IT HAS AN INTERFACE TO EXPORT 



— 101 



I 



DISPLAY DEVICE RECEIVES EXPORT OF 
INTERFACE NOTICE FROM THE DEVICE 
AND SIGNALS BACK TO BEGIN TRANSMITTING 



— 102 



I 



REMOTE DEVICE BROADCASTS PROGRAM 
OBJECTS FOR ITS INTERFACE 



I 



I 



I 



USER INTERACTS WITH REMOTE 
DEVICE S GUI ON DISPLAY DEVICE 



I 



I 



I 



— 103 



DISPLAY DEVICE RECEIVES PROGRAM OBJECTS 
FROM REMOTE DEVICE AND DISPLAYS OBJECT 
ON THE DISPLAY SCREEN 



— 104 



DISPLAY DEVICE INVOKES REMOTE DEVICE'S 

USER INTERFACE PROGRAM OBJECT WHEN 
REMOTE DEVICE DISPLAY OBJECT SELECTED 



— 105 



— 106 



CONTROL METHOD INVOKED ON CORRECT 
REMOTE DEVICE AS INDICATED BY REMOTE 
NAME SERVER, TO CONTROL REMOTE DEVICE 



107 



REMOTE DEVICE CHANGES IN RESPONSE TO 
CONTROL METHOD INVOCATION 



— 108 



RETURN VALUES, IF ANY, ARE BROADCAST BACK 
TO THE DISPLAY DEVICE 



— 109 



Figure 20 



47 



EP 0 626 635 A2 



FINITE STATE MACHINE 
FOR THE REQUESTING SIDE OF A 
REMOTE METHOD INVOCATION 




Figure 21a 



48 



EP 0 626 635 A2 



FINITE STATE MACHINE 
FOR THE RECEIVING SIDE 
OF A REMOTE METHOD 




49 




EP 0 626 635 A2 










EVOCATION 
MENTSARE 
WO PACKET 
ER502 




JFFERGETS 
ITTED 503 


METHOD 11 
AND ARGU 
WRITTEN I] 
BUFF 




PACKET Bl 
TRANSM 





50 



EP 0 626 635 A2 



155 



THERMOSTAT 



424 



— (memory) 



_| 420 \ 



1 



EMITTER 



421 



RECEIVER 



162 




Figure 22a 



Figure 22b 



155 



THERMOSTAT 



— (memory) 



I 



420 \ 

emitter] 



421 



RECEIVER 




424 



Figure 22c 



155 



THERMOSTAT 



424 



— (memory) 



1 



420 \ 



EMITTER 



421 



RECEIVER 




Figure 22d 



51 




Figure 23a 



353 




Figure 23b 

353 




Figure 23c 



353 




421 363 



Figure 23d 

52 




EP 0 626 635 A2 



REMOTE DEVICE'S USER INTERFACE AND CONTROL 
PROGRAM CODE TRANSFERRED TO 
DISPLAY DEVICE 



— 124 



I 



DISPLAY DEVICE INVOKES USER 
INTERFACE PROGRAM CODE 



— 125 



I 



DISPLAY DEVICE DISPLAYS REMOTE DEVICE S 
GRAPHICAL USER INTERFACE 



— 126 



I 



USER INTERACTS WITH DEVICE'S GUI ON DISPLAY 
DEVICE, WHICH CAUSES A CONTROL METHOD 
TO BE INVOKED IN THE DISPLAY DEVICE 



— 127 



I 



METHOD INVOCATION RESOLVES THE USER'S 
ACTIONS INTO A CONTROL CODE FOR 
THE REMOTE DEVICE 



— 128 



I 



CONTROL CODE IS TRANSMITTED TO REMOTE 
DEVICE VIA SOME COMMUNICATION MEANS 



— 129 



I 



REMOTE DEVICE CHANGES IN RESPONSE TO 
THE CONTROL CODE EXECUTION 



I 



—130 



RETURN VALUES, IF ANY, ARE BROADCAST BACK 
TO THE DISPLAY DEVICE 



—131 



Figure 24 



53 



This Page Blank (uspto) 



(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(88) Date of publication A3: 

12.02.1997 Bulletin 1997/07 

(43) Date of publication A2: 

30.11.1994 Bulletin 1994/48 

(21 ) Application number: 94302626.0 

(22) Date of filing: 13.04.1994 



(id EP 0 626 635 A3 

EUROPEAN PATENT APPLICATION 

(51) int. CI 6 : G06F 3/00, G06F 3/033 



(84) Designated Contracting States: 
DE FR GB NL 

(30) Priority: 24.05.1993 US 67574 

(71) Applicant: SUN MICROSYSTEMS, INC. 
Mountain View, CA 94043 (US) 

(72) Inventors: 

• Naughton, Patrick J. 

Palo Alto, California 94036 (US) 

• Warth, Chris 

San Francisco, California 94110 (US) 

• Clanton, Charles H., Ill 

San Francisco, California 94117 (US) 



• Palrang, Joseph M. 
Sunnyvale, California 94086 (US) 

* Gosling, James A. 
Woodside,CA.94062 (US) 

♦ Frank, Edward H. 

Portola Valley, California 94028 (US) 

• LaVallee, David A. 

Palo Alto, California 94301 (US) 

* Sheridan, R. Michael 
Oakton, Virginia 22124 (US) 

(74) Representative: Wombwell, Francis 
Potts, Kerr & Co. 
15, Hamilton Square 
Birkenhead Merseyside L41 6BR (GB) 



CO 
< 

in 

CO 
CO 

CD 
CM 
CD 

o 

Q. 
LU 



(54) Improved graphical user interface with method and apparatus for interfacing to remote 
devices 

(57) An intuitive graphical user interface is based 
upon a geographic map structure, and includes a sys- 
tem for controlling remote external electronic devices. In 
the defined graphical user interface, each Space of the 
geographic map structure is rendered on a touch screen 
display as a graphic image of a geographic space. 
Within each Space are colored cartoon-like icons called 
"objects" which can be Selected and manipulated by the 
user. Certain objects, referred to as portals, transport 
the user from one Space to another Space when 
Selected. Other objects, referred to as buttons, perform 
associated actions or functions when Selected. The 
graphical user interface is displayed on a hand-held dis- 
play device used to control remote devices. Each 
remote electronic device transmits a user interface pro- 
gram object that defines a graphical user interface to the 
display device. The hand-held display devices displays 
the graphical user interface associated with the remote 
device when a user selects an icon associated with the 
remote device on the display device's touch screen dis- 
play. 




Figure la 



Printed by Rank Xerox (UK) Business Services 
2.13 13/3.4 



EP 0 626 635 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 94 30 2626 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication* where appropriate, 
of relevant passages 



Relevant 

to claim 



CLASSIFICATION OF THE 
APPLICATION (Int.CI.5) 



Y 
X 



Y 
A 



US 5 072 412 A (HENDERSON JR D AUSTIN ET 
AL) 10 December 1991 



1,2,4,5, 
7,16, 
42-44 



G06F3/00 
G06F3/033 



* column 


5, 


line 1 - 


column 6, line 23 * 


* column 


14, 


1 ine 


21 


- line 47; figures 


3,4 * 










* column 


16, 


1 ine 


66 


- column 17, line 24; 


figure 5 








- line 22; figure 14 


* column 
* 


27, 


1 ine 


15 


* column 


34, 


1 ine 


55 


- column 36, line 5 * 


* column 


44, 


1 ine 


48 


- line 64 * 



EP 0 439 873 A (IBM ; QUEEN MARY AND 
WESTFIELD COLLE (GB)) 7 August 1991 

* column 1, line 26 - line 37 * 

* column 2, line 51 - column 3, line 9 * 

* column 6, line 9 - line 23 * 

* column 7, line 5 - line 19 * 

* column 9, line 19 - column 10, line 53; 
figures 1,2,4 * 

IBM TECHNICAL DISCLOSURE BULLETIN, 

vol. 36, no. 4, April 1993, NEW YORK, US, 

pages 215-216, XP002012167 

ANONYMOUS: "Animated Conmand Execution 

Icon" 

* the whole document * 



EP 0 438 017 A (IBM) 24 July 1991 

* column 2, line 36 - column 3, line 29; 

figures 1-6 * 

-/-- 



H» pp 



■ ih i u p u 



i u p f ar all do 



10 

1,2,8,9 



TECHNICAL FIELDS 
SEARCHED (Int.C1.S) 



G06F 



18 



10 
1 



8 
s 



EM ace of tcarch 

THE HAGUE 



Date at conpletloa ef the tearch 

30 August 1996 



Nygren, P 



3 
§ 

O 



CATEGORY OF CITED DOCUMENTS 

X : particularly relevant if taken alone 

Y : particularly relevant if combined with another 

document of the same category 
A : technological background 
O : non-written disclosure 
P : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document cited in the application 
L : document dted for other reasons 

& : member of the same patent family, corresponding 
document 



2 



EP 0 626 635 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 94 30 2626 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 

to I 



CLASSIFICATION OF THE 
APPLICATION (IntXLS) 



UIST. PROCEEDINGS OF THE ACM SIGGRAPH 
SYMPOSIUM ON USER INTERFACE SOFTWARE AND 
TECHNOLOGY, 

13 - 15 November 1989, WILLIAMSBURG, VA, 
USA, 

pages 10-18, XP000600624 
ROBERTSON ET AL. : "The Cognitive 
Coprocessor Architecture for Interactive 
User Interfaces" 

* the whole document * 

IBM TECHNICAL DISCLOSURE BULLETIN, 
vol. 34, no. 5, 1 October 1991, 
pages 24-26, XP000189643 "SPATIAL USER 
INTERFACE 1 ' 

* the whole document * 

COMMUNICATIONS OF THE ASSOCIATION FOR 

COMPUTING MACHINERY, 

vol. 36, no. 4, 1 April 1993, 

pages 57-71, XP000355422 

ROBERTSON G G ET AL: "INFORMATION 

VISUALIZATION USING 3D INTERACTIVE 

ANIMATION" 

* the whole document * 

EP 0 524 103 A (INST PERSONALIZED 
INFORMATION) 20 January 1993 

* page 4, line 48 - page 5, line 39; 
figures 2,3,9 * 



TECHNICAL FIELDS 
SEARCHED (Int. CI. 5) 



i Lp w t haj hum di a n w up f ei ai l Uuiim 



Place of i 

THE HAGUE 



Date of cuapiettoa of the tearcfc 

30 August 1996 



Nygren, P 



CATEGORY OF CITED DOCUMENTS 



X : particularly relevant if takes aJone 

Y : particularly relevant rf combined with another 

document of the same category 
A : technological background 
O : oon -writ ten disclosure 
P : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document cited in the application 
L : document cited for other reasons 

A : member of the same patent family, corresponding 
document 



3 



EP 0 626 635 A3 




European Patent 
Office 

EP 94 30 2626.0 



CLAIMS INCURRING FEES 



The present European patent application comprised at tne time of filing more than ten claims. 

□ 



All claims fees have been paid witnin me prescribed time limit, "he present European searcn report ha* been 
crawn uo for all claims. 



□ 
□ 



Only pan of tne claims fees have been paid within the prescribed time limit The present European search 
report has ceen drawn up for tne first ten claims and for tnose claims for which claims fees have been paid, 

namely claims: 

No claims fees nave been oaid within the prescribed time limit. The oresent European searcn report has been 
drawn uo for me first ten claims. 



LACK OF UNITY OF INVENTION 



The Searcn Ofvision considers that tne present European patent application does not comply with the requirement of unity of 

invention and relates to several inventions or groups of inventions. 

namely: 



See Sheet B, 



□ 
□ 



Ail tunner search fees nave been paid witnin (he fixed time limit. The present European search reoort has 
been drawn up for all claims. 

Onty part of the further search fees have oeen paid within tne fixed time limit. The present European searcn 
report has been drawn uo for those pans or the European patent application which relate to the inventions in 
respect of wnicn searcn fees nave been paid. 

namely claims: 

|Xj None of tne funher searcn fees has Oeen paid within tne fixed time limit. 7he present Eurooean searcn report 
nas oeen arawn uo for those parts of the European patent application wnicn relate to the invention first 
mentionea in tne claims. 



namely claims 



: 1-10, 16-18, 42 



4 



EP 0 626 635 A3 



J 



European Patent 
Office 



EP94302626. 0-SHEET B. 



1. Claims 1-10,16-18,42: Method of providing a GUI 

2. Claims 11-15: Method of displaying graphical objects 
for selection 

3. Claim 19: Method for grabbing a graphic object 

4. Claims 20-35: Apparatus for controlling remote devices 
and method of obtaining a GUI display drive 

5. Claims 36-41: Method of copying and moving objects in 
a GUI 



LACK OF UNITY OF INVENTION 



The Sear eft OMaion considers that me present European patent application does not oomph/ with <ne requirement of unity of 

invention and relates 00 several inventions or groups of inventions. 

nameiy: 



5 



Page Blank (uspto) 



